[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:14 dist-test-slave-87nc krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:14 dist-test-slave-87nc krb5kdc[399](info): set up 2 sockets
Jun 20 14:20:14 dist-test-slave-87nc krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:20:15.964581 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:16 dist-test-slave-87nc krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:16.085536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33799
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:16.197764 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:16.198114 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:16.198200 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:16.202196 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:16.202297 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:16.202337 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:16.202373 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:16.202411 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:16.207646 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:16.208746 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:16.209800 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:20:16.218602 412 server_base.cc:1034] running on GCE node
W20260620 14:20:16.218537 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:16.218554 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:16.218554 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:16.219413 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:16.220537 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:16.221725 412 hybrid_clock.cc:648] HybridClock initialized: now 1781965216221682 us; error 54 us; skew 500 ppm
Jun 20 14:20:16 dist-test-slave-87nc krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:16.224381 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:16.225276 412 webserver.cc:460] Webserver started at http://127.0.98.62:39733/ using document root <none> and password file <none>
I20260620 14:20:16.225826 412 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:16.225912 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:16.226135 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:16.228191 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "1e299b4fc16040d0a6d66c0a62f8e795"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "f282c3e54754c30990fc30a68eff94b3"
server_key_iv: "882e30d453f38cd3449b75401656e612"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.228744 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e299b4fc16040d0a6d66c0a62f8e795"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "f282c3e54754c30990fc30a68eff94b3"
server_key_iv: "882e30d453f38cd3449b75401656e612"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.232362 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:20:16.234828 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.236083 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:16.236228 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "1e299b4fc16040d0a6d66c0a62f8e795"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "f282c3e54754c30990fc30a68eff94b3"
server_key_iv: "882e30d453f38cd3449b75401656e612"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.236341 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:16.251163 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:16.254117 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:16.254339 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:16.261407 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33799
I20260620 14:20:16.261430 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33799 every 8 connection(s)
I20260620 14:20:16.262403 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:16.265103 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:16.270300 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795: Bootstrap starting.
I20260620 14:20:16.271952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 412
I20260620 14:20:16.272101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:16.273177 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:16.273672 392 external_mini_cluster.cc:1428] Setting key d8a8e9cf6d7ee923bad61a8ca4d5be99
I20260620 14:20:16.274066 480 log.cc:824] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:16.276235 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795: No bootstrap required, opened a new log
I20260620 14:20:16.280179 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e299b4fc16040d0a6d66c0a62f8e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } }
I20260620 14:20:16.280421 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:16.280500 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e299b4fc16040d0a6d66c0a62f8e795, State: Initialized, Role: FOLLOWER
I20260620 14:20:16.280925 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e299b4fc16040d0a6d66c0a62f8e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } }
I20260620 14:20:16.281073 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:16.281157 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:16.281271 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:16.282173 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e299b4fc16040d0a6d66c0a62f8e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } }
I20260620 14:20:16.282505 480 leader_election.cc:302] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e299b4fc16040d0a6d66c0a62f8e795; no voters:
I20260620 14:20:16.282845 480 leader_election.cc:288] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:16.282912 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:16.283085 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [term 1 LEADER]: Becoming Leader. State: Replica: 1e299b4fc16040d0a6d66c0a62f8e795, State: Running, Role: LEADER
I20260620 14:20:16.283421 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e299b4fc16040d0a6d66c0a62f8e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } }
I20260620 14:20:16.284072 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:16.285527 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e299b4fc16040d0a6d66c0a62f8e795. Latest consensus state: current_term: 1 leader_uuid: "1e299b4fc16040d0a6d66c0a62f8e795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e299b4fc16040d0a6d66c0a62f8e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } } }
I20260620 14:20:16.285744 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:16.286633 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e299b4fc16040d0a6d66c0a62f8e795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e299b4fc16040d0a6d66c0a62f8e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } } }
I20260620 14:20:16.286764 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795 [sys.catalog]: This master's current role is: LEADER
W20260620 14:20:16.289125 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:20:16.289222 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:20:16.289340 497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:16.290153 497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:20:16 dist-test-slave-87nc krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:16.296103 497 catalog_manager.cc:1347] Generated new cluster ID: 234327df5e364824b794578aa51346b6
I20260620 14:20:16.296197 497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:16.300444 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:16.277000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.0.98.62:33799)
0620 14:20:16.277462 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:16.277482 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:16.278686 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 14:20:16.281606 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:16.281620 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:16.281917 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:16.282238 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:16.285615 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.287158 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.288839 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.289537 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.289712 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:16.294314 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:16.294322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:16.294338 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:16.294371 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:16.296733 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:16.297333 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:16.297339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:16.297345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:16.297493 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:16.297851 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:16.297855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:16.297857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:16.298371 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:16.298576 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:16.299152 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 14:20:16.299450 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":136,"threads_started":1}
I20260620 14:20:16.308308 497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:16.309171 497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:16.316905 497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e299b4fc16040d0a6d66c0a62f8e795: Generated new TSK 0
I20260620 14:20:16.317710 497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:20:16.350713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:16.460372 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:16.460646 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:16.460691 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:16.464645 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:16.464706 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:16.464766 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:16.469900 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:16.470924 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:16.471954 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:20:16.481665 506 server_base.cc:1034] running on GCE node
W20260620 14:20:16.481630 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:16.481655 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:16.481630 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:16.482321 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:16.482999 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:16.484175 506 hybrid_clock.cc:648] HybridClock initialized: now 1781965216484159 us; error 28 us; skew 500 ppm
Jun 20 14:20:16 dist-test-slave-87nc krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:16.487015 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:16.487919 506 webserver.cc:460] Webserver started at http://127.0.98.1:39587/ using document root <none> and password file <none>
I20260620 14:20:16.488451 506 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:16.488538 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:16.488770 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:16.490743 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "4d0b4c4c169f4923807f4535b7bf68fe"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "dc202f718056610539210ca58810f6b9"
server_key_iv: "42cb0c0f43c94e1bce4cfb8d954f1f04"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.491304 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4d0b4c4c169f4923807f4535b7bf68fe"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "dc202f718056610539210ca58810f6b9"
server_key_iv: "42cb0c0f43c94e1bce4cfb8d954f1f04"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.494931 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 14:20:16.497270 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.498430 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:16.498556 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "4d0b4c4c169f4923807f4535b7bf68fe"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "dc202f718056610539210ca58810f6b9"
server_key_iv: "42cb0c0f43c94e1bce4cfb8d954f1f04"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.498674 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:16.508204 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:16.510913 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:16.511114 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:16.511732 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:16.512614 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:16.512684 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.512753 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:16.512815 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.522408 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35191
I20260620 14:20:16.522431 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35191 every 8 connection(s)
I20260620 14:20:16.523283 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:16.526575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 506
I20260620 14:20:16.526700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:16.527010 392 external_mini_cluster.cc:1428] Setting key f60a055baa7c4b2f130b268fa23adc93
Jun 20 14:20:16 dist-test-slave-87nc krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:20:16.539018 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:16.525201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54645 (local address 127.0.98.62:33799)
0620 14:20:16.525367 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:16.525372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:16.526341 (+ 969us) server_negotiation.cc:407] Connection header received
0620 14:20:16.528055 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:16.528058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:16.528113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:16.528226 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:16.529909 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.530426 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.532280 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.532508 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.532589 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:16.535432 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:16.535436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:16.535439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:16.535469 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:16.536897 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:16.537417 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:16.537420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:16.537422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:16.537509 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:16.537857 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:16.537862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:16.537864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:16.538103 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:16.538211 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:16.538728 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 14:20:16.538855 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:20:16.540198 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:16.525615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33799 (local address 127.0.98.1:54645)
0620 14:20:16.526083 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:16.526128 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:16.527815 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:16.528435 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:16.528446 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:16.528974 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:16.529736 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:16.529758 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.530555 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.530561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:16.532138 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:16.532153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.532667 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.532674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:16.532908 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:16.533544 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:16.533571 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:16.535271 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:16.537013 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:16.537019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:16.537035 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:16.537308 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:16.537610 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:16.537613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:16.537615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:16.537761 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:16.538205 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:16.538211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:16.538479 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 14:20:16.538687 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 14:20:16.539089 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":105,"threads_started":1}
I20260620 14:20:16.541499 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33799
I20260620 14:20:16.541782 635 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:16.542392 635 heartbeater.cc:511] Master 127.0.98.62:33799 requested a full tablet report, sending...
I20260620 14:20:16.544102 444 ts_manager.cc:194] Registered new tserver with Master: 4d0b4c4c169f4923807f4535b7bf68fe (127.0.98.1:35191)
I20260620 14:20:16.545163 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:20:16.554935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:16.666867 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:16.667164 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:16.667256 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:16.671298 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:16.671396 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:16.671486 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:16.676764 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:16.677850 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:16.678850 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:16.688380 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:16.688403 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:16.688544 640 server_base.cc:1034] running on GCE node
W20260620 14:20:16.688428 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:16.688916 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:16.689502 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:16.690701 640 hybrid_clock.cc:648] HybridClock initialized: now 1781965216690688 us; error 27 us; skew 500 ppm
Jun 20 14:20:16 dist-test-slave-87nc krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:16.693506 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:16.694278 640 webserver.cc:460] Webserver started at http://127.0.98.2:40725/ using document root <none> and password file <none>
I20260620 14:20:16.694780 640 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:16.694836 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:16.694990 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:16.696966 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "ea210784d8894225b98b0051a43daafa"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "85ddd106d7153693c6f257c2b1286712"
server_key_iv: "7fc3197c12b66914d1f9386369648dde"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.697438 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ea210784d8894225b98b0051a43daafa"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "85ddd106d7153693c6f257c2b1286712"
server_key_iv: "7fc3197c12b66914d1f9386369648dde"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.700862 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:20:16.703227 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.704433 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:16.704530 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "ea210784d8894225b98b0051a43daafa"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "85ddd106d7153693c6f257c2b1286712"
server_key_iv: "7fc3197c12b66914d1f9386369648dde"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.704609 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:16.729782 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:16.732784 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:16.732935 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:16.733448 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:16.734213 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:16.734257 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.734295 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:16.734361 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.743856 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36885
I20260620 14:20:16.743880 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36885 every 8 connection(s)
I20260620 14:20:16.744735 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:16.751004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 640
I20260620 14:20:16.751094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:16.751356 392 external_mini_cluster.cc:1428] Setting key aff7fb2cfd3f1cb9ecd87de89b024d38
Jun 20 14:20:16 dist-test-slave-87nc krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:16.759080 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:16.746513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51165 (local address 127.0.98.62:33799)
0620 14:20:16.746706 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:16.746711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:16.747516 (+ 805us) server_negotiation.cc:407] Connection header received
0620 14:20:16.748866 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:16.748869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:16.748916 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:16.748989 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:16.750100 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.750610 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.751897 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.752236 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.752341 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:16.754768 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:16.754774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:16.754778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:16.754810 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:16.756916 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:16.757458 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:16.757462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:16.757464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:16.757556 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:16.757888 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:16.757894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:16.757899 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:16.758168 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:16.758283 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:16.758747 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 14:20:16.758891 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 14:20:16.759867 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:16.746913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33799 (local address 127.0.98.2:51165)
0620 14:20:16.747368 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:16.747411 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:16.748675 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:16.749146 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:16.749153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:16.749509 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:16.749965 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:16.749978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.750722 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.750724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:16.751745 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:16.751755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.752378 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.752381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:16.752519 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:16.753198 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:16.753214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:16.754608 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:16.757056 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:16.757062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:16.757081 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:16.757355 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:16.757650 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:16.757652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:16.757654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:16.757791 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:16.758276 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:16.758280 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:16.758506 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 14:20:16.758660 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 14:20:16.758920 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":148,"threads_started":1}
I20260620 14:20:16.761047 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33799
I20260620 14:20:16.761284 769 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:20:16.761823 769 heartbeater.cc:511] Master 127.0.98.62:33799 requested a full tablet report, sending...
I20260620 14:20:16.762869 444 ts_manager.cc:194] Registered new tserver with Master: ea210784d8894225b98b0051a43daafa (127.0.98.2:36885)
I20260620 14:20:16.763422 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:20:16.778543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:16.891395 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:16.891714 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:16.891856 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:16.895951 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:16.896054 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:16.896158 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:16.901518 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:16.902602 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:16.903671 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:16.913091 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:16.913130 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:16.913300 774 server_base.cc:1034] running on GCE node
W20260620 14:20:16.913358 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:16.913802 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:16.914423 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:16.915609 774 hybrid_clock.cc:648] HybridClock initialized: now 1781965216915591 us; error 30 us; skew 500 ppm
Jun 20 14:20:16 dist-test-slave-87nc krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:16.918218 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:16.919041 774 webserver.cc:460] Webserver started at http://127.0.98.3:35653/ using document root <none> and password file <none>
I20260620 14:20:16.919590 774 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:16.919692 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:16.919945 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:16.921909 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "16e3e1991858460fa33c82fd070eb150"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "9409db6c18d251832454254268ecf46d"
server_key_iv: "6b60c24b69598880756227985a1d92af"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.922488 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "16e3e1991858460fa33c82fd070eb150"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "9409db6c18d251832454254268ecf46d"
server_key_iv: "6b60c24b69598880756227985a1d92af"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.926277 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 14:20:16.928593 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.929745 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:20:16.929881 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "16e3e1991858460fa33c82fd070eb150"
format_stamp: "Formatted at 2026-06-20 14:20:16 on dist-test-slave-87nc"
server_key: "9409db6c18d251832454254268ecf46d"
server_key_iv: "6b60c24b69598880756227985a1d92af"
server_key_version: "encryptionkey@0"
I20260620 14:20:16.929986 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:16.956426 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:16.959335 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:16.959583 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:16.960232 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:16.961087 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:16.961159 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.961227 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:16.961273 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:16.970710 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36093
I20260620 14:20:16.970727 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36093 every 8 connection(s)
I20260620 14:20:16.971571 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:16.974709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 774
I20260620 14:20:16.974831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:16.975121 392 external_mini_cluster.cc:1428] Setting key be23f14632f87ba90e7e0f6842c6de47
Jun 20 14:20:16 dist-test-slave-87nc krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:16.985240 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:16.973384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38089 (local address 127.0.98.62:33799)
0620 14:20:16.973518 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:16.973522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:16.974314 (+ 792us) server_negotiation.cc:407] Connection header received
0620 14:20:16.976121 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:16.976124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:16.976187 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:16.976281 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:16.977467 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.978007 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.979126 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.979354 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.979469 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:16.981818 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:16.981822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:16.981825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:16.981857 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:16.983066 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:16.983577 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:16.983582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:16.983583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:16.983669 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:16.984075 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:16.984078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:16.984080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:16.984294 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:16.984416 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:16.984913 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 14:20:16.985064 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:20:16.986053 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:16.973739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33799 (local address 127.0.98.3:38089)
0620 14:20:16.974160 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:16.974207 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:16.975877 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:16.976461 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:16.976468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:16.976859 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:16.977326 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:16.977340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.978136 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.978140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:16.978999 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:16.979008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:16.979502 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:16.979506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:16.979669 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:16.980265 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:16.980280 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:16.981642 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:16.983183 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:16.983191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:16.983207 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:16.983469 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:16.983806 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:16.983809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:16.983811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:16.983985 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:16.984423 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:16.984431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:16.984667 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 14:20:16.984855 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 14:20:16.985165 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260620 14:20:16.987252 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33799
I20260620 14:20:16.987514 903 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:16.988077 903 heartbeater.cc:511] Master 127.0.98.62:33799 requested a full tablet report, sending...
I20260620 14:20:16.989118 443 ts_manager.cc:194] Registered new tserver with Master: 16e3e1991858460fa33c82fd070eb150 (127.0.98.3:36093)
I20260620 14:20:16.989821 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38089
I20260620 14:20:17.001698 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:20:17 dist-test-slave-87nc krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965217, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:20:17 dist-test-slave-87nc krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965217, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:17.027508 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:17.018406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.0.98.1:35191)
0620 14:20:17.018664 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:17.018707 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:17.018736 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:20:17.018946 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:17.018952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:17.019148 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:17.019301 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:17.019991 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:17.020752 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:17.021606 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:17.022015 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:17.022140 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:17.024487 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:17.024495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:17.024506 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:17.024530 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:17.025947 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:17.026397 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:17.026403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:17.026404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:17.026485 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:17.026713 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:17.026716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:17.026717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:17.026933 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:17.027093 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:17.027244 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:20:17.027366 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
W20260620 14:20:17.028496 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35818
Jun 20 14:20:17 dist-test-slave-87nc krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:20:17 dist-test-slave-87nc krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:17.052944 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:17.045129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.0.98.1:35191)
0620 14:20:17.045279 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:17.045283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:17.045418 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:20:17.045517 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:17.045519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:17.045563 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:17.045634 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:17.046300 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:17.046804 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:17.047824 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:17.048065 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:17.048145 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:17.050443 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:17.050447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:17.050449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:17.050470 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:17.051620 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:17.052052 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:17.052055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:17.052056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:17.052123 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:17.052334 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:17.052336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:17.052337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:17.052472 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:17.052560 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:17.052735 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:20:17.052824 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:20:17.053943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 506
I20260620 14:20:17.060252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 640
I20260620 14:20:17.068058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 774
I20260620 14:20:17.073359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 412
2026-06-20T14:20:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2184 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:17 dist-test-slave-87nc krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:17 dist-test-slave-87nc krb5kdc[917](info): set up 2 sockets
Jun 20 14:20:17 dist-test-slave-87nc krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 14:20:18.111466 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:18 dist-test-slave-87nc krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:18.230243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37415
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:18.338457 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:18.338712 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:18.338752 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:18.342782 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:18.342833 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:18.342854 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:18.342869 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:18.342885 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:18.348203 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:18.349229 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:18.350241 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:18.358578 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:18.358613 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:18.358665 930 server_base.cc:1034] running on GCE node
W20260620 14:20:18.358613 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:18.359323 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:18.360430 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:18.361588 930 hybrid_clock.cc:648] HybridClock initialized: now 1781965218361570 us; error 30 us; skew 500 ppm
Jun 20 14:20:18 dist-test-slave-87nc krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:18.364238 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:18.365012 930 webserver.cc:460] Webserver started at http://127.0.98.62:35775/ using document root <none> and password file <none>
I20260620 14:20:18.365561 930 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:18.365623 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:18.365787 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:18.367705 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "a1ce8d92e31a42eca02ef07340dc19cc"
format_stamp: "Formatted at 2026-06-20 14:20:18 on dist-test-slave-87nc"
server_key: "49c44209c8b5c5320ec4d9f5cd0a631f"
server_key_iv: "d67a6238bf3d8a82bf3008dcb0924dcc"
server_key_version: "encryptionkey@0"
I20260620 14:20:18.368261 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1ce8d92e31a42eca02ef07340dc19cc"
format_stamp: "Formatted at 2026-06-20 14:20:18 on dist-test-slave-87nc"
server_key: "49c44209c8b5c5320ec4d9f5cd0a631f"
server_key_iv: "d67a6238bf3d8a82bf3008dcb0924dcc"
server_key_version: "encryptionkey@0"
I20260620 14:20:18.371649 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:20:18.373943 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:18.375092 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:18.375214 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "a1ce8d92e31a42eca02ef07340dc19cc"
format_stamp: "Formatted at 2026-06-20 14:20:18 on dist-test-slave-87nc"
server_key: "49c44209c8b5c5320ec4d9f5cd0a631f"
server_key_iv: "d67a6238bf3d8a82bf3008dcb0924dcc"
server_key_version: "encryptionkey@0"
I20260620 14:20:18.375298 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:18.388656 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:18.391148 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:18.391295 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:18.397830 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37415
I20260620 14:20:18.397917 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37415 every 8 connection(s)
I20260620 14:20:18.398790 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:18.401661 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:18.406327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 930
I20260620 14:20:18.406424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:18.406702 392 external_mini_cluster.cc:1428] Setting key 63ee6823e29fef1824eef3dfe7204935
I20260620 14:20:18.407362 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc: Bootstrap starting.
I20260620 14:20:18.410019 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:18.410912 998 log.cc:824] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:18.412954 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc: No bootstrap required, opened a new log
Jun 20 14:20:18 dist-test-slave-87nc krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:18.417032 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ce8d92e31a42eca02ef07340dc19cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260620 14:20:18.417339 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:18.417423 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ce8d92e31a42eca02ef07340dc19cc, State: Initialized, Role: FOLLOWER
I20260620 14:20:18.417920 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ce8d92e31a42eca02ef07340dc19cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260620 14:20:18.418102 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:18.418180 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:18.418278 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:18.419092 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ce8d92e31a42eca02ef07340dc19cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260620 14:20:18.419472 998 leader_election.cc:302] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1ce8d92e31a42eca02ef07340dc19cc; no voters:
I20260620 14:20:18.419827 998 leader_election.cc:288] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:18.419893 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:18.420095 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [term 1 LEADER]: Becoming Leader. State: Replica: a1ce8d92e31a42eca02ef07340dc19cc, State: Running, Role: LEADER
I20260620 14:20:18.420380 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ce8d92e31a42eca02ef07340dc19cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260620 14:20:18.421093 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:18.422152 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1ce8d92e31a42eca02ef07340dc19cc. Latest consensus state: current_term: 1 leader_uuid: "a1ce8d92e31a42eca02ef07340dc19cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ce8d92e31a42eca02ef07340dc19cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } } }
I20260620 14:20:18.422327 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:18.422717 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:18.423198 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:18.408133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.0.98.62:37415)
0620 14:20:18.408578 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:18.408596 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:18.408642 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:20:18.409453 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:18.409475 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:18.409893 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:18.410269 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:18.410892 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:18.412296 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:18.413162 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:18.413614 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:18.413782 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:18.416603 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:18.416612 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:18.416641 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:18.416677 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:18.419157 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:18.419750 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:18.419759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:18.419786 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:18.420034 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:18.420448 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:18.420454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:18.420459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:18.421014 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:18.421298 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:18.421496 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:20:18.421800 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260620 14:20:18.422437 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1ce8d92e31a42eca02ef07340dc19cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ce8d92e31a42eca02ef07340dc19cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } } }
I20260620 14:20:18.424522 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:18.425832 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:18.431782 1012 catalog_manager.cc:1347] Generated new cluster ID: 3a9745f277694b769517ba9bb7b61e87
I20260620 14:20:18.431846 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:18.451566 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:18.452636 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:18.461395 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1ce8d92e31a42eca02ef07340dc19cc: Generated new TSK 0
I20260620 14:20:18.462123 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260620 14:20:19.647455 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 1.182s user 0.000s sys 0.003s
I20260620 14:20:19.648027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:19.761804 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:19.762045 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:19.762115 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:19.766244 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:19.766304 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:19.766403 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:19.771593 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:19.772732 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:19.773861 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:19.783804 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:19.783881 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:19.783882 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:19.784154 1024 server_base.cc:1034] running on GCE node
I20260620 14:20:19.784546 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:19.785184 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:19.786370 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781965219786354 us; error 26 us; skew 500 ppm
Jun 20 14:20:19 dist-test-slave-87nc krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:19.789036 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:19.789898 1024 webserver.cc:460] Webserver started at http://127.0.98.1:44921/ using document root <none> and password file <none>
I20260620 14:20:19.790482 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:19.790568 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:19.790802 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:19.792939 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "b9c7024ffb6f480fa024752e07f38f59"
format_stamp: "Formatted at 2026-06-20 14:20:19 on dist-test-slave-87nc"
server_key: "41ebd0085a67661015aa4996e0036277"
server_key_iv: "c6018c76938579674e59f1d41c54241f"
server_key_version: "encryptionkey@0"
I20260620 14:20:19.793574 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b9c7024ffb6f480fa024752e07f38f59"
format_stamp: "Formatted at 2026-06-20 14:20:19 on dist-test-slave-87nc"
server_key: "41ebd0085a67661015aa4996e0036277"
server_key_iv: "c6018c76938579674e59f1d41c54241f"
server_key_version: "encryptionkey@0"
I20260620 14:20:19.797353 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:20:19.799726 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:19.801019 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:19.801162 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "b9c7024ffb6f480fa024752e07f38f59"
format_stamp: "Formatted at 2026-06-20 14:20:19 on dist-test-slave-87nc"
server_key: "41ebd0085a67661015aa4996e0036277"
server_key_iv: "c6018c76938579674e59f1d41c54241f"
server_key_version: "encryptionkey@0"
I20260620 14:20:19.801273 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:19.837050 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:19.840032 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:19.840261 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:19.840875 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:19.841704 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:19.841778 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:19.841848 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:19.841905 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:19.850965 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33655
I20260620 14:20:19.850986 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33655 every 8 connection(s)
I20260620 14:20:19.851958 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:19.854420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 1024
I20260620 14:20:19.854524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:19.854794 392 external_mini_cluster.cc:1428] Setting key 6bc1fa22704d4c3a3f8063bcca29485d
Jun 20 14:20:19 dist-test-slave-87nc krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:20:19.867410 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:19.853694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60871 (local address 127.0.98.62:37415)
0620 14:20:19.853979 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:19.853982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:19.854819 (+ 837us) server_negotiation.cc:407] Connection header received
0620 14:20:19.857099 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:19.857102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:19.857164 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:19.857280 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:19.858913 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:19.859854 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:19.861082 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:19.861540 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:19.861672 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:19.864134 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:19.864141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:19.864146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:19.864178 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:19.865616 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:19.866060 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:19.866063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:19.866064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:19.866152 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:19.866454 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:19.866456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:19.866457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:19.866656 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:19.866732 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:19.867183 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 14:20:19.867279 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260620 14:20:19.868304 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:19.854106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37415 (local address 127.0.98.1:60871)
0620 14:20:19.854676 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:19.854717 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:19.856880 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:19.857490 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:19.857501 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:19.858027 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:19.858755 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:19.858771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:19.859978 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:19.859981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:19.860952 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:19.860961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:19.861703 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:19.861706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:19.861843 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:19.862477 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:19.862493 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:19.863963 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:19.865720 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:19.865724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:19.865736 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:19.865975 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:19.866241 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:19.866243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:19.866244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:19.866375 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:19.866745 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:19.866749 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:19.866974 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 14:20:19.867134 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 14:20:19.867385 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":146,"threads_started":1}
I20260620 14:20:19.869510 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37415
I20260620 14:20:19.869717 1153 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:19.870163 1153 heartbeater.cc:511] Master 127.0.98.62:37415 requested a full tablet report, sending...
I20260620 14:20:19.871805 962 ts_manager.cc:194] Registered new tserver with Master: b9c7024ffb6f480fa024752e07f38f59 (127.0.98.1:33655)
I20260620 14:20:19.873116 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:20:20.876004 1153 heartbeater.cc:503] Master 127.0.98.62:37415 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260620 14:20:21.037482 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.181s user 0.001s sys 0.002s
I20260620 14:20:21.038247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:21.156057 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:21.156332 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:21.156420 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:21.160832 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:21.160916 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:21.161058 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:21.166735 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:21.168002 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:21.169123 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:21.179800 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:21.180006 1159 server_base.cc:1034] running on GCE node
W20260620 14:20:21.179841 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:21.179800 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:21.180585 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:21.181284 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:21.182448 1159 hybrid_clock.cc:648] HybridClock initialized: now 1781965221182430 us; error 30 us; skew 500 ppm
Jun 20 14:20:21 dist-test-slave-87nc krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:21.185438 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:21.186290 1159 webserver.cc:460] Webserver started at http://127.0.98.2:45167/ using document root <none> and password file <none>
I20260620 14:20:21.186815 1159 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:21.186873 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:21.187031 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:21.189055 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "9f6e336afb8a4235905ccbde8c8b8948"
format_stamp: "Formatted at 2026-06-20 14:20:21 on dist-test-slave-87nc"
server_key: "d2eb7d92133cbe82e079b36d3caab939"
server_key_iv: "335e74d6a3227544c4a16b2be7f5fc71"
server_key_version: "encryptionkey@0"
I20260620 14:20:21.189569 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9f6e336afb8a4235905ccbde8c8b8948"
format_stamp: "Formatted at 2026-06-20 14:20:21 on dist-test-slave-87nc"
server_key: "d2eb7d92133cbe82e079b36d3caab939"
server_key_iv: "335e74d6a3227544c4a16b2be7f5fc71"
server_key_version: "encryptionkey@0"
I20260620 14:20:21.193248 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:20:21.196039 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:21.197513 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:20:21.197659 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "9f6e336afb8a4235905ccbde8c8b8948"
format_stamp: "Formatted at 2026-06-20 14:20:21 on dist-test-slave-87nc"
server_key: "d2eb7d92133cbe82e079b36d3caab939"
server_key_iv: "335e74d6a3227544c4a16b2be7f5fc71"
server_key_version: "encryptionkey@0"
I20260620 14:20:21.197784 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:21.215603 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:21.219069 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:21.219218 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:21.219782 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:21.220602 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:21.220655 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:21.220693 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:21.220708 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:21.229676 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38015
I20260620 14:20:21.229697 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38015 every 8 connection(s)
I20260620 14:20:21.230564 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:21.234849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 1159
I20260620 14:20:21.234972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:21.235261 392 external_mini_cluster.cc:1428] Setting key f8c157b8391694a8ca53994716809313
Jun 20 14:20:21 dist-test-slave-87nc krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:20:21.245965 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:21.232287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33777 (local address 127.0.98.62:37415)
0620 14:20:21.232517 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:21.232521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:21.233240 (+ 719us) server_negotiation.cc:407] Connection header received
0620 14:20:21.234532 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:21.234534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:21.234582 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:21.234667 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:21.236264 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:21.236966 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:21.238726 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:21.239138 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:21.239259 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:21.241938 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:21.241945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:21.241951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:21.241987 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:21.243801 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:21.244403 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:21.244408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:21.244410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:21.244505 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:21.244853 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:21.244857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:21.244859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:21.245087 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:21.245212 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:21.245724 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 14:20:21.245827 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260620 14:20:21.246834 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:21.232682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37415 (local address 127.0.98.2:33777)
0620 14:20:21.233085 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:21.233130 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:21.234332 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:21.234900 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:21.234910 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:21.235438 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:21.236122 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:21.236139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:21.237102 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:21.237106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:21.238594 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:21.238605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:21.239294 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:21.239297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:21.239451 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:21.240132 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:21.240151 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:21.241748 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:21.243910 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:21.243915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:21.243928 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:21.244201 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:21.244604 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:21.244606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:21.244607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:21.244751 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:21.245191 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:21.245196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:21.245498 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 14:20:21.245678 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 14:20:21.245951 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
I20260620 14:20:21.248142 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37415
I20260620 14:20:21.248395 1288 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:21.248894 1288 heartbeater.cc:511] Master 127.0.98.62:37415 requested a full tablet report, sending...
I20260620 14:20:21.249965 962 ts_manager.cc:194] Registered new tserver with Master: 9f6e336afb8a4235905ccbde8c8b8948 (127.0.98.2:38015)
I20260620 14:20:21.250568 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260620 14:20:21.946712 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.710s user 0.002s sys 0.001s
I20260620 14:20:21.947266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:22.057227 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:22.057490 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:22.057534 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:22.061607 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:22.061667 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:22.061725 1294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:22.066821 1294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:22.067781 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:22.068729 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:22.078040 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:22.078068 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:22.078081 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:22.078202 1294 server_base.cc:1034] running on GCE node
I20260620 14:20:22.078706 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:22.079665 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:22.080848 1294 hybrid_clock.cc:648] HybridClock initialized: now 1781965222080806 us; error 34 us; skew 500 ppm
Jun 20 14:20:22 dist-test-slave-87nc krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:22.083386 1294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:22.084251 1294 webserver.cc:460] Webserver started at http://127.0.98.3:35053/ using document root <none> and password file <none>
I20260620 14:20:22.084784 1294 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:22.084877 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:22.085090 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:22.087023 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "7e8310f527a84907859571996d709d3f"
format_stamp: "Formatted at 2026-06-20 14:20:22 on dist-test-slave-87nc"
server_key: "6734ee4f76c6282bcb9f5b04d534430b"
server_key_iv: "5683cf033e9e60f220f2a3798e8f04af"
server_key_version: "encryptionkey@0"
I20260620 14:20:22.087576 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7e8310f527a84907859571996d709d3f"
format_stamp: "Formatted at 2026-06-20 14:20:22 on dist-test-slave-87nc"
server_key: "6734ee4f76c6282bcb9f5b04d534430b"
server_key_iv: "5683cf033e9e60f220f2a3798e8f04af"
server_key_version: "encryptionkey@0"
I20260620 14:20:22.091109 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:20:22.093601 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:22.094883 1294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:20:22.095029 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "7e8310f527a84907859571996d709d3f"
format_stamp: "Formatted at 2026-06-20 14:20:22 on dist-test-slave-87nc"
server_key: "6734ee4f76c6282bcb9f5b04d534430b"
server_key_iv: "5683cf033e9e60f220f2a3798e8f04af"
server_key_version: "encryptionkey@0"
I20260620 14:20:22.095134 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:22.108971 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:22.111689 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:22.111896 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:22.112564 1294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:22.113386 1294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:22.113432 1294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:22.113508 1294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:22.113544 1294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:22.122347 1294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35441
I20260620 14:20:22.122364 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35441 every 8 connection(s)
I20260620 14:20:22.123274 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:20:22 dist-test-slave-87nc krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:22.133474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 1294
I20260620 14:20:22.133605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:22.133886 392 external_mini_cluster.cc:1428] Setting key 4d1ec4655cec0201e1b5712eff1e6921
I20260620 14:20:22.137372 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.125104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56401 (local address 127.0.98.62:37415)
0620 14:20:22.125336 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.125339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.126071 (+ 732us) server_negotiation.cc:407] Connection header received
0620 14:20:22.127461 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.127466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.127524 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.127614 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:22.128809 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.129369 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.130828 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.131063 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.131176 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.133604 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:22.133608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:22.133611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:22.133641 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:22.135429 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.136023 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.136030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.136034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.136125 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.136476 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.136478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.136479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.136653 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:22.136729 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.137159 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.137249 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260620 14:20:22.138223 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.125494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37415 (local address 127.0.98.3:56401)
0620 14:20:22.125920 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:22.125964 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:22.127250 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:22.127804 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:22.127811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:22.128194 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:22.128655 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.128668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.129503 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.129507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.130692 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.130700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.131211 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.131215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.131381 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.132015 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:22.132030 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:22.133428 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:22.135543 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:22.135548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:22.135562 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:22.135925 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:22.136228 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:22.136234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:22.136240 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:22.136379 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:22.136739 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:22.136744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:22.136958 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 14:20:22.137100 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 14:20:22.137351 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":118,"threads_started":1}
I20260620 14:20:22.139310 1423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37415
I20260620 14:20:22.139556 1423 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:22.140134 1423 heartbeater.cc:511] Master 127.0.98.62:37415 requested a full tablet report, sending...
I20260620 14:20:22.141177 962 ts_manager.cc:194] Registered new tserver with Master: 7e8310f527a84907859571996d709d3f (127.0.98.3:35441)
I20260620 14:20:22.141723 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56401
I20260620 14:20:22.147838 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:20:22.157671 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.151106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.0.98.62:37415)
0620 14:20:22.151241 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.151244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.151382 (+ 138us) server_negotiation.cc:407] Connection header received
0620 14:20:22.151465 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.151467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.151502 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.151568 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:22.152297 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.152798 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.153551 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.153766 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.153837 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.155039 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:22.155041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:22.155044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:22.155066 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:22.156370 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.156735 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.156738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.156739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.156807 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.157013 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.157015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.157017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.157199 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:22.157297 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.157462 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.157550 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:20:22.162187 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:22.165216 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:22.179955 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.174409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.0.98.3:35441)
0620 14:20:22.174868 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.174880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.174902 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:20:22.174975 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.174982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.175184 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.175336 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:22.175893 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.176841 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.178344 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.179205 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.179337 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.179532 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.179659 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.179757 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":93,"threads_started":1}
I20260620 14:20:22.179941 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.173581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35441 (local address 127.0.0.1:48098)
0620 14:20:22.174475 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:22.174490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:22.174581 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:22.175316 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:22.175318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:22.175348 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:22.175539 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.175547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.176966 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.176972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.178189 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.178198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.179349 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.179353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.179457 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.179500 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:20:22.179623 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:20:22.179748 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":39,"spinlock_wait_cycles":4480,"thread_start_us":108,"threads_started":1}
I20260620 14:20:22.179992 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.173837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33655 (local address 127.0.0.1:47576)
0620 14:20:22.175164 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:22.175176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:22.175261 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:22.175928 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:22.175934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:22.175956 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:22.176046 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.176051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.177641 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.177644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.178708 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.178715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.179633 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.179636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.179726 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.179745 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:20:22.179824 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:20:22.179881 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"thread_start_us":83,"threads_started":1}
I20260620 14:20:22.180224 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.173541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38015 (local address 127.0.0.1:58074)
0620 14:20:22.174313 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:22.174364 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:22.174576 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:22.175306 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:22.175312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:22.175347 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:22.175539 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.175547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.177054 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.177057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.178411 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.178421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.179599 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.179603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.179692 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.179714 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:20:22.179848 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:20:22.179922 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":126,"spinlock_wait_cycles":5248,"thread_start_us":129,"threads_started":1}
I20260620 14:20:22.180292 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.173653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.0.98.2:38015)
0620 14:20:22.174067 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.174075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.174458 (+ 383us) server_negotiation.cc:407] Connection header received
0620 14:20:22.174768 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.174774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.174996 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.175112 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:22.175667 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.176923 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.178529 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.179266 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.179358 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.179538 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.180002 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.180116 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":119,"threads_started":1}
I20260620 14:20:22.181182 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.174778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.0.98.1:33655)
0620 14:20:22.175494 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.175500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.175518 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:20:22.175573 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.175576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.175771 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.175947 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:22.176171 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.177290 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.178856 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.179495 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.180715 (+ 1220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.180821 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.180858 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.180977 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":90,"threads_started":1}
I20260620 14:20:22.183475 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 771ee080bac442e28498551a6be1fdf4 (DEFAULT_TABLE table=test-table [id=bac32448bca8446ba12492e728829c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:22.183475 1357 tablet_service.cc:1467] Processing CreateTablet for tablet 771ee080bac442e28498551a6be1fdf4 (DEFAULT_TABLE table=test-table [id=bac32448bca8446ba12492e728829c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:22.183735 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 771ee080bac442e28498551a6be1fdf4 (DEFAULT_TABLE table=test-table [id=bac32448bca8446ba12492e728829c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:22.184530 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771ee080bac442e28498551a6be1fdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:22.184545 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771ee080bac442e28498551a6be1fdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:22.184563 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771ee080bac442e28498551a6be1fdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:22.190353 1442 tablet_bootstrap.cc:492] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948: Bootstrap starting.
I20260620 14:20:22.192078 1444 tablet_bootstrap.cc:492] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f: Bootstrap starting.
I20260620 14:20:22.192893 1442 tablet_bootstrap.cc:654] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:22.193846 1442 log.cc:824] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:22.194468 1444 tablet_bootstrap.cc:654] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:22.194475 1443 tablet_bootstrap.cc:492] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59: Bootstrap starting.
I20260620 14:20:22.195713 1444 log.cc:824] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:22.196098 1442 tablet_bootstrap.cc:492] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948: No bootstrap required, opened a new log
I20260620 14:20:22.196292 1442 ts_tablet_manager.cc:1397] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:20:22.196296 1443 tablet_bootstrap.cc:654] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:22.196906 1443 log.cc:824] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:22.197638 1444 tablet_bootstrap.cc:492] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f: No bootstrap required, opened a new log
I20260620 14:20:22.197811 1444 ts_tablet_manager.cc:1397] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:20:22.198596 1443 tablet_bootstrap.cc:492] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59: No bootstrap required, opened a new log
I20260620 14:20:22.198807 1443 ts_tablet_manager.cc:1397] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 14:20:22.201095 1442 raft_consensus.cc:348] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } }
I20260620 14:20:22.201380 1442 raft_consensus.cc:374] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:22.201440 1442 raft_consensus.cc:729] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6e336afb8a4235905ccbde8c8b8948, State: Initialized, Role: FOLLOWER
I20260620 14:20:22.201668 1444 raft_consensus.cc:348] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } }
I20260620 14:20:22.201924 1444 raft_consensus.cc:374] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:22.202013 1444 raft_consensus.cc:729] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8310f527a84907859571996d709d3f, State: Initialized, Role: FOLLOWER
I20260620 14:20:22.201985 1442 consensus_queue.cc:260] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } }
I20260620 14:20:22.202440 1444 consensus_queue.cc:260] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } }
I20260620 14:20:22.202689 1288 heartbeater.cc:503] Master 127.0.98.62:37415 was elected leader, sending a full tablet report...
I20260620 14:20:22.203050 1442 ts_tablet_manager.cc:1428] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:20:22.203347 1444 ts_tablet_manager.cc:1428] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:20:22.203699 1423 heartbeater.cc:503] Master 127.0.98.62:37415 was elected leader, sending a full tablet report...
I20260620 14:20:22.204375 1443 raft_consensus.cc:348] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } }
I20260620 14:20:22.204632 1443 raft_consensus.cc:374] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:22.204700 1443 raft_consensus.cc:729] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c7024ffb6f480fa024752e07f38f59, State: Initialized, Role: FOLLOWER
I20260620 14:20:22.205123 1443 consensus_queue.cc:260] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } }
I20260620 14:20:22.206019 1443 ts_tablet_manager.cc:1428] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260620 14:20:22.231930 1289 tablet.cc:2369] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:22.354612 1154 tablet.cc:2369] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:22.374303 1424 tablet.cc:2369] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:22.462260 1449 raft_consensus.cc:482] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:22.462467 1449 raft_consensus.cc:504] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } }
I20260620 14:20:22.463573 1449 leader_election.cc:288] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6e336afb8a4235905ccbde8c8b8948 (127.0.98.2:38015), b9c7024ffb6f480fa024752e07f38f59 (127.0.98.1:33655)
I20260620 14:20:22.467756 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.463992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39545 (local address 127.0.98.1:33655)
0620 14:20:22.464122 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.464130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.464150 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:20:22.464327 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.464330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.464373 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.464463 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:22.464737 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.465334 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.466588 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.467091 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.467199 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.467261 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.467521 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.467610 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:20:22.467803 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.463890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33655 (local address 127.0.98.3:39545)
0620 14:20:22.464037 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:22.464055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:22.464187 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:22.464471 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:22.464473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:22.464488 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:22.464610 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.464616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.465447 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.465453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.466462 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.466471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.467221 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.467227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.467367 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.467406 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:20:22.467487 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:20:22.467578 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 14:20:22.468215 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.463985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54357 (local address 127.0.98.2:38015)
0620 14:20:22.464112 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.464117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.464490 (+ 373us) server_negotiation.cc:407] Connection header received
0620 14:20:22.464668 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.464670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.464711 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.464794 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:22.465061 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.465705 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.466952 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.467515 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.467611 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.467664 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.468008 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.468090 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:20:22.468554 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.463890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38015 (local address 127.0.98.3:54357)
0620 14:20:22.464378 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:22.464412 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:22.464535 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:22.464805 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:22.464807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:22.464822 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:22.464952 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.464957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.465814 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.465819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.466816 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:22.466828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.467721 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.467724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:22.467847 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.467874 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:20:22.468352 (+ 478us) client_negotiation.cc:240] Negotiation successful
0620 14:20:22.468423 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":125,"spinlock_wait_cycles":3712,"thread_start_us":145,"threads_started":1}
I20260620 14:20:22.468537 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771ee080bac442e28498551a6be1fdf4" candidate_uuid: "7e8310f527a84907859571996d709d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c7024ffb6f480fa024752e07f38f59" is_pre_election: true
I20260620 14:20:22.468912 1107 raft_consensus.cc:2454] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e8310f527a84907859571996d709d3f in term 0.
I20260620 14:20:22.469283 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771ee080bac442e28498551a6be1fdf4" candidate_uuid: "7e8310f527a84907859571996d709d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6e336afb8a4235905ccbde8c8b8948" is_pre_election: true
I20260620 14:20:22.469458 1310 leader_election.cc:302] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e8310f527a84907859571996d709d3f, b9c7024ffb6f480fa024752e07f38f59; no voters:
I20260620 14:20:22.469606 1242 raft_consensus.cc:2454] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e8310f527a84907859571996d709d3f in term 0.
I20260620 14:20:22.469754 1449 raft_consensus.cc:2781] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:22.469856 1449 raft_consensus.cc:482] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:22.469913 1449 raft_consensus.cc:3037] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:22.470966 1449 raft_consensus.cc:504] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } }
I20260620 14:20:22.471302 1449 leader_election.cc:288] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [CANDIDATE]: Term 1 election: Requested vote from peers 9f6e336afb8a4235905ccbde8c8b8948 (127.0.98.2:38015), b9c7024ffb6f480fa024752e07f38f59 (127.0.98.1:33655)
I20260620 14:20:22.471560 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771ee080bac442e28498551a6be1fdf4" candidate_uuid: "7e8310f527a84907859571996d709d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6e336afb8a4235905ccbde8c8b8948"
I20260620 14:20:22.471722 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771ee080bac442e28498551a6be1fdf4" candidate_uuid: "7e8310f527a84907859571996d709d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c7024ffb6f480fa024752e07f38f59"
I20260620 14:20:22.471730 1242 raft_consensus.cc:3037] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:22.471861 1107 raft_consensus.cc:3037] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:22.472950 1242 raft_consensus.cc:2454] T 771ee080bac442e28498551a6be1fdf4 P 9f6e336afb8a4235905ccbde8c8b8948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8310f527a84907859571996d709d3f in term 1.
I20260620 14:20:22.473014 1107 raft_consensus.cc:2454] T 771ee080bac442e28498551a6be1fdf4 P b9c7024ffb6f480fa024752e07f38f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8310f527a84907859571996d709d3f in term 1.
I20260620 14:20:22.473285 1310 leader_election.cc:302] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e8310f527a84907859571996d709d3f, b9c7024ffb6f480fa024752e07f38f59; no voters:
I20260620 14:20:22.473483 1449 raft_consensus.cc:2781] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:22.473650 1449 raft_consensus.cc:686] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [term 1 LEADER]: Becoming Leader. State: Replica: 7e8310f527a84907859571996d709d3f, State: Running, Role: LEADER
I20260620 14:20:22.473990 1449 consensus_queue.cc:237] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } }
I20260620 14:20:22.477227 961 catalog_manager.cc:5495] T 771ee080bac442e28498551a6be1fdf4 P 7e8310f527a84907859571996d709d3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e8310f527a84907859571996d709d3f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7e8310f527a84907859571996d709d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6e336afb8a4235905ccbde8c8b8948" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e8310f527a84907859571996d709d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9c7024ffb6f480fa024752e07f38f59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33655 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:20:22 dist-test-slave-87nc krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:22.546761 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.538109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.0.98.1:33655)
0620 14:20:22.538265 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.538268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.538316 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:20:22.538421 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.538423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.538463 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.538532 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:22.539207 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.539699 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.540554 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.540769 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.540846 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.543171 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:22.543182 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:22.543196 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:22.543219 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:22.544965 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.545379 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.545389 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.545394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.545486 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.545731 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.545736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.545741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.546064 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:22.546288 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.546378 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.546506 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:20:22 dist-test-slave-87nc krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965222, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:20:22 dist-test-slave-87nc krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965222, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:22.659314 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.650238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.0.98.1:33655)
0620 14:20:22.650395 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.650401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.650524 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:20:22.650637 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.650640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.650686 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.650762 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:22.651835 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.652356 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.653643 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.653869 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.653953 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.656370 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:22.656374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:22.656376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:22.656401 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:22.657685 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.658195 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.658198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.658199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.658276 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.658496 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.658498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.658499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.658662 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:22.658759 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.658990 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.659089 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:20:22 dist-test-slave-87nc krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:20:22 dist-test-slave-87nc krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:22.747745 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:22.736512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.0.98.1:33655)
0620 14:20:22.736688 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:22.736696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:22.736888 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:20:22.737035 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:22.737039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:22.737106 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:22.737217 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:22.738335 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.739314 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.740741 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:22.741215 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:22.741360 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:22.744143 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:22.744150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:22.744154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:22.744192 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:22.745940 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.746331 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.746337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.746341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.746439 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:22.746700 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:22.746705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:22.746708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:22.746969 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:22.747128 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:22.747339 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:20:22.747491 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 14:20:22.749593 1067 tablet_service.cc:2908] Rejecting scan request for tablet 771ee080bac442e28498551a6be1fdf4: Uninitialized: clean time has not yet been initialized
I20260620 14:20:22.751675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 1024
I20260620 14:20:22.758754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 1159
I20260620 14:20:22.764393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 1294
I20260620 14:20:22.769558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 930
2026-06-20T14:20:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5696 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:24 dist-test-slave-87nc krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:24 dist-test-slave-87nc krb5kdc[1466](info): set up 2 sockets
Jun 20 14:20:24 dist-test-slave-87nc krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260620 14:20:25.095501 392 mini_kdc.cc:121] Time spent starting KDC: real 2.303s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:25 dist-test-slave-87nc krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:25.210619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:25.332711 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:25.332983 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:25.333025 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:25.337056 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:25.337121 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:20:25.337141 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:25.337162 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:25.337181 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:25.337199 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:25.342461 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46049
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:25.343415 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:25.344398 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:25.352602 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:25.352612 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:25.352725 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:25.352975 1479 server_base.cc:1034] running on GCE node
I20260620 14:20:25.353628 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:25.354701 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:25.355839 1479 hybrid_clock.cc:648] HybridClock initialized: now 1781965225355806 us; error 45 us; skew 500 ppm
Jun 20 14:20:25 dist-test-slave-87nc krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:25.358417 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:25.359340 1479 webserver.cc:460] Webserver started at http://127.0.98.62:42577/ using document root <none> and password file <none>
I20260620 14:20:25.359961 1479 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:25.360052 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:25.360280 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:25.362303 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "bb0b111c3b5b4631ba8c0c919380a543"
format_stamp: "Formatted at 2026-06-20 14:20:25 on dist-test-slave-87nc"
server_key: "e78e61a4b1a18bd5ed03ca79fb0d814f"
server_key_iv: "4cc068d2c42cc852dde91f57d4c2d399"
server_key_version: "encryptionkey@0"
I20260620 14:20:25.362876 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb0b111c3b5b4631ba8c0c919380a543"
format_stamp: "Formatted at 2026-06-20 14:20:25 on dist-test-slave-87nc"
server_key: "e78e61a4b1a18bd5ed03ca79fb0d814f"
server_key_iv: "4cc068d2c42cc852dde91f57d4c2d399"
server_key_version: "encryptionkey@0"
I20260620 14:20:25.366536 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 14:20:25.369009 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:25.370204 1479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:25.370302 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "bb0b111c3b5b4631ba8c0c919380a543"
format_stamp: "Formatted at 2026-06-20 14:20:25 on dist-test-slave-87nc"
server_key: "e78e61a4b1a18bd5ed03ca79fb0d814f"
server_key_iv: "4cc068d2c42cc852dde91f57d4c2d399"
server_key_version: "encryptionkey@0"
I20260620 14:20:25.370384 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:25.386065 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:25.388834 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:25.388986 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:25.397087 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46049
I20260620 14:20:25.397123 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46049 every 8 connection(s)
I20260620 14:20:25.398034 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:25.400753 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:25.405872 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543: Bootstrap starting.
I20260620 14:20:25.406170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 1479
I20260620 14:20:25.406266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:25.406510 392 external_mini_cluster.cc:1428] Setting key cda44b8e9b8ba1ffc729e053d127ab65
I20260620 14:20:25.408474 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:25.409205 1557 log.cc:824] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:25.410979 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543: No bootstrap required, opened a new log
I20260620 14:20:25.414577 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b111c3b5b4631ba8c0c919380a543" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260620 14:20:25.414871 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:25.414934 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0b111c3b5b4631ba8c0c919380a543, State: Initialized, Role: FOLLOWER
Jun 20 14:20:25 dist-test-slave-87nc krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:25.415428 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b111c3b5b4631ba8c0c919380a543" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260620 14:20:25.415611 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:25.415684 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:25.415835 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:25.417007 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b111c3b5b4631ba8c0c919380a543" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260620 14:20:25.417347 1557 leader_election.cc:302] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0b111c3b5b4631ba8c0c919380a543; no voters:
I20260620 14:20:25.417683 1557 leader_election.cc:288] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:25.417753 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:25.417950 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [term 1 LEADER]: Becoming Leader. State: Replica: bb0b111c3b5b4631ba8c0c919380a543, State: Running, Role: LEADER
I20260620 14:20:25.418262 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b111c3b5b4631ba8c0c919380a543" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260620 14:20:25.418964 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:25.419392 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0b111c3b5b4631ba8c0c919380a543. Latest consensus state: current_term: 1 leader_uuid: "bb0b111c3b5b4631ba8c0c919380a543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b111c3b5b4631ba8c0c919380a543" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } } }
I20260620 14:20:25.419572 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:25.421054 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:25.407814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.0.98.62:46049)
0620 14:20:25.408249 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:25.408263 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:25.408297 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:20:25.408970 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:25.408989 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:25.409371 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:25.409696 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:25.410523 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.411550 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.412490 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.412951 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.413135 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:25.415838 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:25.415844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:25.415865 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:25.415892 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:25.417770 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.418248 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.418254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.418275 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.418441 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.418721 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.418726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.418727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.419180 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:25.419385 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:25.419660 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:20:25.420068 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":130,"threads_started":1}
I20260620 14:20:25.422781 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb0b111c3b5b4631ba8c0c919380a543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b111c3b5b4631ba8c0c919380a543" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } } }
I20260620 14:20:25.422940 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543 [sys.catalog]: This master's current role is: LEADER
W20260620 14:20:25.423964 1577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:20:25.424042 1577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:20:25.424119 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:25.424758 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:25.429606 1567 catalog_manager.cc:1347] Generated new cluster ID: ca52d8ec22df45869e9d27783b097090
I20260620 14:20:25.429667 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:25.453624 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:25.454797 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:25.461513 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543: Generated new TSK 0
I20260620 14:20:25.462235 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:20:25.490042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46049
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:25.599913 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:25.600148 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:25.600219 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:25.604164 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:25.604214 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:25.604280 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:20:25.604367 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:25.609444 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:25.610437 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:25.611421 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:25.621527 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:25.621533 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:25.621626 1583 server_base.cc:1034] running on GCE node
W20260620 14:20:25.621539 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:25.622118 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:25.622751 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:25.623910 1583 hybrid_clock.cc:648] HybridClock initialized: now 1781965225623895 us; error 27 us; skew 500 ppm
Jun 20 14:20:25 dist-test-slave-87nc krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:25.626641 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:25.627475 1583 webserver.cc:460] Webserver started at http://127.0.98.1:42321/ using document root <none> and password file <none>
I20260620 14:20:25.628105 1583 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:25.628170 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:25.628410 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:25.630424 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "83fe91090fe145fcb38d1dc05f05ec51"
format_stamp: "Formatted at 2026-06-20 14:20:25 on dist-test-slave-87nc"
server_key: "87921fc2629f6f4775e791c8fe82de25"
server_key_iv: "e4be1587993f500ee2036c2a2bd36f62"
server_key_version: "encryptionkey@0"
I20260620 14:20:25.630993 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83fe91090fe145fcb38d1dc05f05ec51"
format_stamp: "Formatted at 2026-06-20 14:20:25 on dist-test-slave-87nc"
server_key: "87921fc2629f6f4775e791c8fe82de25"
server_key_iv: "e4be1587993f500ee2036c2a2bd36f62"
server_key_version: "encryptionkey@0"
I20260620 14:20:25.634624 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 14:20:25.637017 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:25.638206 1583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:25.638325 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "83fe91090fe145fcb38d1dc05f05ec51"
format_stamp: "Formatted at 2026-06-20 14:20:25 on dist-test-slave-87nc"
server_key: "87921fc2629f6f4775e791c8fe82de25"
server_key_iv: "e4be1587993f500ee2036c2a2bd36f62"
server_key_version: "encryptionkey@0"
I20260620 14:20:25.638427 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:25.681879 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:25.684818 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:25.685016 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:25.686488 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:25.686537 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:25.686612 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:25.686652 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:20:25 dist-test-slave-87nc krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:25.697535 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45721
I20260620 14:20:25.697621 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45721 every 8 connection(s)
I20260620 14:20:25.698704 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:25.701083 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:25.687338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34757 (local address 127.0.98.62:46049)
0620 14:20:25.687461 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:25.687465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:25.688490 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 14:20:25.689481 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:25.689485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:25.689542 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:25.689623 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:25.690890 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.691617 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.693420 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.693833 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.693947 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:25.697644 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:25.697648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:25.697651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:25.697681 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:25.699206 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.699847 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.699850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.699851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.699931 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.700217 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.700219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.700220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.700383 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:25.700455 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:25.700871 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 14:20:25.700969 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:20:25.701826 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:25.687888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46049 (local address 127.0.98.1:34757)
0620 14:20:25.688345 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:25.688387 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:25.689221 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:25.689983 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:25.689989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:25.690292 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:25.690734 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:25.690748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.691742 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.691748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:25.693297 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:25.693312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.694003 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.694010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:25.694267 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:25.695549 (+ 1282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:25.695572 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:25.697460 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:25.699339 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:25.699346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:25.699362 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:25.699685 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:25.699999 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:25.700001 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:25.700003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:25.700132 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:25.700468 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:25.700472 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:25.700674 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 14:20:25.700814 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 14:20:25.701036 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":134,"threads_started":1}
I20260620 14:20:25.703125 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46049
I20260620 14:20:25.703281 1714 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:25.703750 1714 heartbeater.cc:511] Master 127.0.98.62:46049 requested a full tablet report, sending...
I20260620 14:20:25.705297 1510 ts_manager.cc:194] Registered new tserver with Master: 83fe91090fe145fcb38d1dc05f05ec51 (127.0.98.1:45721)
I20260620 14:20:25.706373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 1583
I20260620 14:20:25.706476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:25.706652 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34757
I20260620 14:20:25.706764 392 external_mini_cluster.cc:1428] Setting key adb835e848b5456d5fcdbbe2d4a8f40f
I20260620 14:20:25.714038 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:25.705208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50127 (local address 127.0.98.62:46049)
0620 14:20:25.705326 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:25.705330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:25.705476 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:20:25.705619 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:25.705621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:25.705658 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:25.705739 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:25.706187 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.706751 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.707753 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.708105 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.708219 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:25.710946 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:25.710949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:25.710951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:25.710972 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:25.712414 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.712907 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.712909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.712910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.712979 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.713291 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.713293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.713294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.713446 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:25.713517 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:25.713841 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:20:25.713927 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:20:25.714226 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:25.705131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46049 (local address 127.0.98.1:50127)
0620 14:20:25.705405 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:25.705423 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:25.705515 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:25.705755 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:25.705758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:25.705949 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:25.706036 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:25.706042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.706872 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.706875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:25.707633 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:25.707642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.709855 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.709858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:25.709936 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:25.710445 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:25.710456 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:25.710809 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:25.712524 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:25.712529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:25.712533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:25.712807 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:25.713070 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:25.713075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:25.713078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:25.713190 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:25.713543 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:25.713547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:25.713667 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:20:25.713801 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:20:25.713972 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:20:25.734217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46049
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:25.841698 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:25.842015 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:25.842067 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:25.846195 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:25.846256 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:25.846338 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:20:25.846402 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:25.851500 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:25.852566 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:25.853578 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:25.863183 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:25.863201 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:25.863201 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:25.863306 1723 server_base.cc:1034] running on GCE node
I20260620 14:20:25.863816 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:25.864531 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:25.865780 1723 hybrid_clock.cc:648] HybridClock initialized: now 1781965225865749 us; error 46 us; skew 500 ppm
Jun 20 14:20:25 dist-test-slave-87nc krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:25.868605 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:25.869424 1723 webserver.cc:460] Webserver started at http://127.0.98.2:33875/ using document root <none> and password file <none>
I20260620 14:20:25.869975 1723 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:25.870064 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:25.870290 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:25.872365 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "696424886b604f299c62f879ede74dd7"
format_stamp: "Formatted at 2026-06-20 14:20:25 on dist-test-slave-87nc"
server_key: "3b4e2bffc10e58d1d1d8beff91637f0d"
server_key_iv: "3729e584a94df3c3ca5e9c5171ad92e6"
server_key_version: "encryptionkey@0"
I20260620 14:20:25.872952 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "696424886b604f299c62f879ede74dd7"
format_stamp: "Formatted at 2026-06-20 14:20:25 on dist-test-slave-87nc"
server_key: "3b4e2bffc10e58d1d1d8beff91637f0d"
server_key_iv: "3729e584a94df3c3ca5e9c5171ad92e6"
server_key_version: "encryptionkey@0"
I20260620 14:20:25.876773 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:20:25.878988 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:25.880223 1723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:25.880360 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "696424886b604f299c62f879ede74dd7"
format_stamp: "Formatted at 2026-06-20 14:20:25 on dist-test-slave-87nc"
server_key: "3b4e2bffc10e58d1d1d8beff91637f0d"
server_key_iv: "3729e584a94df3c3ca5e9c5171ad92e6"
server_key_version: "encryptionkey@0"
I20260620 14:20:25.880465 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:25.896837 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:25.899669 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:25.899893 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:25.901258 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:25.901335 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:25.901399 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:25.901450 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:20:25 dist-test-slave-87nc krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:25.912568 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36877
I20260620 14:20:25.912619 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36877 every 8 connection(s)
I20260620 14:20:25.913417 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:25.915176 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:25.902028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57351 (local address 127.0.98.62:46049)
0620 14:20:25.902154 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:25.902166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:25.903186 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 14:20:25.904239 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:25.904243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:25.904309 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:25.904413 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:25.905627 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.906495 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.908137 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.908414 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.908524 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:25.911343 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:25.911348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:25.911352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:25.911386 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:25.913011 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.913536 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.913539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.913540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.913613 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.913983 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.913986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.913988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.914215 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:25.914325 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:25.914890 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 14:20:25.914999 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:20:25.915926 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:25.902593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46049 (local address 127.0.98.2:57351)
0620 14:20:25.903041 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:25.903088 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:25.904005 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:25.904599 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:25.904609 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:25.904991 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:25.905482 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:25.905501 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.906633 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.906638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:25.908019 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:25.908027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.908583 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.908588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:25.908752 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:25.909380 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:25.909408 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:25.911178 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:25.913129 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:25.913135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:25.913150 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:25.913404 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:25.913704 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:25.913709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:25.913711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:25.913873 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:25.914464 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:25.914469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:25.914674 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 14:20:25.914849 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 14:20:25.915136 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260620 14:20:25.917043 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46049
I20260620 14:20:25.917178 1854 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:25.917699 1854 heartbeater.cc:511] Master 127.0.98.62:46049 requested a full tablet report, sending...
I20260620 14:20:25.918712 1510 ts_manager.cc:194] Registered new tserver with Master: 696424886b604f299c62f879ede74dd7 (127.0.98.2:36877)
I20260620 14:20:25.919458 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57351
I20260620 14:20:25.920104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 1723
I20260620 14:20:25.920224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:25.920495 392 external_mini_cluster.cc:1428] Setting key 116401d5eb2472fbfbf294d5bb495527
I20260620 14:20:25.927043 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:25.919302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41135 (local address 127.0.98.62:46049)
0620 14:20:25.919412 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:25.919418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:25.919696 (+ 278us) server_negotiation.cc:407] Connection header received
0620 14:20:25.919871 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:25.919892 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:25.919941 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:25.920017 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:25.920603 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.921326 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.922300 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.922526 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.922632 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:25.923902 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:25.923905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:25.923907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:25.923927 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:25.925413 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.925877 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.925879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.925880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.925949 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:25.926247 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:25.926249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:25.926250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:25.926432 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:25.926508 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:25.926839 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 14:20:25.926932 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:20:25.927208 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:25.919249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46049 (local address 127.0.98.2:41135)
0620 14:20:25.919619 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:25.919633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:25.919729 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:25.920139 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:25.920143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:25.920337 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:25.920477 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:25.920482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.921434 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.921437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:25.922183 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:25.922191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:25.922633 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:25.922636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:25.922723 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:25.923260 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:25.923279 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:25.923752 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:25.925523 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:25.925526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:25.925529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:25.925786 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:25.926039 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:25.926045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:25.926048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:25.926154 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:25.926528 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:25.926533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:25.926654 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:20:25.926781 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:20:25.926952 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:20:25.946689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46049
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:26.054994 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:26.055238 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:26.055284 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:26.059216 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:26.059269 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:26.059304 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:20:26.059368 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:26.064476 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:26.065438 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:26.066398 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:26.075886 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:26.075889 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:26.075882 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:26.076157 1863 server_base.cc:1034] running on GCE node
I20260620 14:20:26.076531 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:26.077138 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:26.078289 1863 hybrid_clock.cc:648] HybridClock initialized: now 1781965226078270 us; error 29 us; skew 500 ppm
Jun 20 14:20:26 dist-test-slave-87nc krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:26.080919 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:26.081733 1863 webserver.cc:460] Webserver started at http://127.0.98.3:37011/ using document root <none> and password file <none>
I20260620 14:20:26.082291 1863 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:26.082377 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:26.082592 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:26.084765 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "5af3cff9f192489296a0ca727e919e43"
format_stamp: "Formatted at 2026-06-20 14:20:26 on dist-test-slave-87nc"
server_key: "5b676b0f3f133cc120ddbd376515b540"
server_key_iv: "2a707fa0ea3c330f181b19fcf39f9f73"
server_key_version: "encryptionkey@0"
I20260620 14:20:26.085321 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5af3cff9f192489296a0ca727e919e43"
format_stamp: "Formatted at 2026-06-20 14:20:26 on dist-test-slave-87nc"
server_key: "5b676b0f3f133cc120ddbd376515b540"
server_key_iv: "2a707fa0ea3c330f181b19fcf39f9f73"
server_key_version: "encryptionkey@0"
I20260620 14:20:26.088865 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:20:26.091143 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:26.092469 1863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:20:26.092602 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "5af3cff9f192489296a0ca727e919e43"
format_stamp: "Formatted at 2026-06-20 14:20:26 on dist-test-slave-87nc"
server_key: "5b676b0f3f133cc120ddbd376515b540"
server_key_iv: "2a707fa0ea3c330f181b19fcf39f9f73"
server_key_version: "encryptionkey@0"
I20260620 14:20:26.092710 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:26.120638 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:26.123718 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:26.123956 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:26.125458 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:26.125531 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:26.125615 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:26.125670 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:20:26 dist-test-slave-87nc krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:26.136366 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43635
I20260620 14:20:26.136438 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43635 every 8 connection(s)
I20260620 14:20:26.137245 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:26.140163 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.126187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56991 (local address 127.0.98.62:46049)
0620 14:20:26.126315 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.126321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.127363 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 14:20:26.128479 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.128482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.128543 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.128627 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:26.129858 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.130766 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.132564 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.132878 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.133001 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.136388 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:26.136393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:26.136396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:26.136431 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:26.138143 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.138649 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.138659 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.138661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.138733 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.139080 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.139083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.139085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.139263 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:26.139379 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.139876 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.139997 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:20:26.140919 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.126766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46049 (local address 127.0.98.3:56991)
0620 14:20:26.127217 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.127258 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.128263 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.128919 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.128926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.129249 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:26.129694 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.129706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.130916 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.130922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.132405 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.132420 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.133046 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.133050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.133258 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.134345 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:26.134361 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:26.136175 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:26.138260 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:26.138266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:26.138280 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:26.138536 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:26.138826 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:26.138830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:26.138832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:26.138978 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:26.139365 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:26.139371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:26.139615 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.139811 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.140135 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":126,"threads_started":1}
I20260620 14:20:26.142066 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46049
I20260620 14:20:26.142220 1994 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:26.142697 1994 heartbeater.cc:511] Master 127.0.98.62:46049 requested a full tablet report, sending...
I20260620 14:20:26.143635 1510 ts_manager.cc:194] Registered new tserver with Master: 5af3cff9f192489296a0ca727e919e43 (127.0.98.3:43635)
I20260620 14:20:26.144282 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56991
I20260620 14:20:26.144631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 1863
I20260620 14:20:26.144744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:26.145035 392 external_mini_cluster.cc:1428] Setting key 714d4125153916eb0af7971d4f3f9f6a
I20260620 14:20:26.146994 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:20:26.152927 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.144400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46449 (local address 127.0.98.62:46049)
0620 14:20:26.144561 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.144565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.145582 (+ 1017us) server_negotiation.cc:407] Connection header received
0620 14:20:26.145784 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.145786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.145848 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.145929 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:26.146416 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.147125 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.148290 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.148513 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.148589 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.149811 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:26.149816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:26.149821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:26.149858 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:26.151255 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.151701 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.151706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.151708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.151839 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.152131 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.152134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.152136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.152307 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:26.152411 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.152661 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.152767 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:20:26.152933 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.144343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46049 (local address 127.0.98.3:46449)
0620 14:20:26.145493 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.145507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.145620 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.145965 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.145968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.146192 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:26.146313 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.146319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.147244 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.147248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.148181 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.148195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.148622 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.148628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.148752 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.149359 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:26.149370 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:26.149678 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:26.151366 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:26.151371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:26.151374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:26.151588 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:26.151929 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:26.151933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:26.151935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:26.152024 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:26.152400 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:26.152403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:26.152501 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.152611 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.152756 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":78,"threads_started":1}
Jun 20 14:20:26 dist-test-slave-87nc krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965226, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:20:26 dist-test-slave-87nc krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965226, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:26.173444 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.165513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.0.98.62:46049)
0620 14:20:26.165664 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.165667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.165681 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:20:26.165723 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.165726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.165764 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.165845 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:26.166569 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.167086 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.167998 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.168307 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.168399 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.170724 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:26.170726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:26.170729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:26.170751 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:26.172032 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.172399 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.172401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.172402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.172470 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.172684 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.172686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.172687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.172841 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:26.172936 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.173185 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.173273 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:20:26.176388 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:26.179334 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:26.193569 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.187745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43635 (local address 127.0.0.1:36044)
0620 14:20:26.188217 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.188270 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.188399 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.189142 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.189147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.189169 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:26.189297 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.189305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.190725 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.190728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.191878 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.191886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.192956 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.192960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.193062 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.193175 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.193279 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.193399 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260620 14:20:26.193750 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.188664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.0.98.1:45721)
0620 14:20:26.188997 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.189012 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.189031 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:20:26.189116 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.189121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.189285 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.189674 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:26.189800 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.190817 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.192417 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.193197 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.193313 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.193456 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.193514 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.193612 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":77,"threads_started":1}
I20260620 14:20:26.193742 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.187845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45721 (local address 127.0.0.1:46526)
0620 14:20:26.188737 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.188752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.188840 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.189427 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.189430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.189447 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:26.189528 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.189534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.190926 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.190932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.192276 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.192287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.193312 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.193315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.193402 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.193424 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.193490 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.193559 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":42,"thread_start_us":106,"threads_started":1}
I20260620 14:20:26.194178 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.188464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36877 (local address 127.0.0.1:58148)
0620 14:20:26.189267 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.189279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.189367 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.189828 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.189831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.189849 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:26.189946 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.189954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.190967 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.190970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.192295 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.192301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.193366 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.193370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.193463 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.193486 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.193551 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.193639 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":68,"thread_start_us":84,"threads_started":1}
I20260620 14:20:26.194615 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.188667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.0.98.2:36877)
0620 14:20:26.189079 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.189092 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.189460 (+ 368us) server_negotiation.cc:407] Connection header received
0620 14:20:26.189538 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.189543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.189693 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.189841 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:26.190059 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.190861 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.192432 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.193041 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.193145 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.193310 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.194340 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.194453 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":106,"threads_started":1}
I20260620 14:20:26.194869 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.187895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.0.98.3:43635)
0620 14:20:26.188356 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.188372 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.188396 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:20:26.188607 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.188616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.188825 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.188975 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:26.189431 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.190607 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.192002 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.192824 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.192934 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.193104 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.194632 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.194740 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":123,"threads_started":1}
I20260620 14:20:26.196902 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 2c53e9e53b704c09a7fbde651823e66a (DEFAULT_TABLE table=test-table [id=e5ba7836e5554c0096518de4d4497f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:26.197052 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 2c53e9e53b704c09a7fbde651823e66a (DEFAULT_TABLE table=test-table [id=e5ba7836e5554c0096518de4d4497f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:26.197472 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 2c53e9e53b704c09a7fbde651823e66a (DEFAULT_TABLE table=test-table [id=e5ba7836e5554c0096518de4d4497f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:26.197971 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c53e9e53b704c09a7fbde651823e66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:26.197973 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c53e9e53b704c09a7fbde651823e66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:26.198313 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c53e9e53b704c09a7fbde651823e66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:26.205034 2017 tablet_bootstrap.cc:492] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: Bootstrap starting.
I20260620 14:20:26.206017 2019 tablet_bootstrap.cc:492] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: Bootstrap starting.
I20260620 14:20:26.207444 2017 tablet_bootstrap.cc:654] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:26.207688 2019 tablet_bootstrap.cc:654] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:26.208261 2017 log.cc:824] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:26.208395 2019 log.cc:824] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:26.208763 2018 tablet_bootstrap.cc:492] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: Bootstrap starting.
I20260620 14:20:26.210457 2017 tablet_bootstrap.cc:492] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: No bootstrap required, opened a new log
I20260620 14:20:26.210457 2019 tablet_bootstrap.cc:492] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: No bootstrap required, opened a new log
I20260620 14:20:26.210464 2018 tablet_bootstrap.cc:654] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:26.210650 2017 ts_tablet_manager.cc:1397] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 14:20:26.210650 2019 ts_tablet_manager.cc:1397] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:20:26.211230 2018 log.cc:824] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:26.213002 2018 tablet_bootstrap.cc:492] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: No bootstrap required, opened a new log
I20260620 14:20:26.213209 2018 ts_tablet_manager.cc:1397] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:20:26.214645 2017 raft_consensus.cc:348] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260620 14:20:26.214645 2019 raft_consensus.cc:348] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260620 14:20:26.214920 2019 raft_consensus.cc:374] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:26.214920 2017 raft_consensus.cc:374] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:26.214977 2017 raft_consensus.cc:729] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696424886b604f299c62f879ede74dd7, State: Initialized, Role: FOLLOWER
I20260620 14:20:26.214977 2019 raft_consensus.cc:729] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af3cff9f192489296a0ca727e919e43, State: Initialized, Role: FOLLOWER
I20260620 14:20:26.215462 2019 consensus_queue.cc:260] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260620 14:20:26.215462 2017 consensus_queue.cc:260] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260620 14:20:26.216394 1994 heartbeater.cc:503] Master 127.0.98.62:46049 was elected leader, sending a full tablet report...
I20260620 14:20:26.216427 1854 heartbeater.cc:503] Master 127.0.98.62:46049 was elected leader, sending a full tablet report...
I20260620 14:20:26.216547 2019 ts_tablet_manager.cc:1428] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260620 14:20:26.217018 2018 raft_consensus.cc:348] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260620 14:20:26.217475 2017 ts_tablet_manager.cc:1428] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260620 14:20:26.217269 2018 raft_consensus.cc:374] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:26.217945 2018 raft_consensus.cc:729] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fe91090fe145fcb38d1dc05f05ec51, State: Initialized, Role: FOLLOWER
I20260620 14:20:26.218420 2018 consensus_queue.cc:260] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260620 14:20:26.219007 1714 heartbeater.cc:503] Master 127.0.98.62:46049 was elected leader, sending a full tablet report...
I20260620 14:20:26.219244 2018 ts_tablet_manager.cc:1428] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
W20260620 14:20:26.388242 1995 tablet.cc:2369] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:26.414584 1855 tablet.cc:2369] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:26.450134 1715 tablet.cc:2369] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:26.450201 2023 raft_consensus.cc:482] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:26.450405 2023 raft_consensus.cc:504] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260620 14:20:26.451506 2023 leader_election.cc:288] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5af3cff9f192489296a0ca727e919e43 (127.0.98.3:43635), 83fe91090fe145fcb38d1dc05f05ec51 (127.0.98.1:45721)
I20260620 14:20:26.455332 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.451912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52373 (local address 127.0.98.1:45721)
0620 14:20:26.452039 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.452044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.452205 (+ 161us) server_negotiation.cc:407] Connection header received
0620 14:20:26.452375 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.452378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.452429 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.452511 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:26.452823 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.453319 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.454385 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.454776 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.454889 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.454953 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.455128 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.455192 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:20:26.455376 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.451912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45151 (local address 127.0.98.3:43635)
0620 14:20:26.452135 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.452140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.452275 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:20:26.452436 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.452444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.452487 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.452571 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:26.452895 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.453385 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.454421 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.454809 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.454927 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.454986 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.455171 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.455240 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 14:20:26.455382 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.451827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45721 (local address 127.0.98.2:52373)
0620 14:20:26.452095 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.452110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.452255 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.452522 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.452525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.452540 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:26.452712 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.452718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.453422 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.453428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.454263 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.454271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.454894 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.454897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.454993 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.455015 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.455113 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.455192 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":50,"thread_start_us":69,"threads_started":1}
I20260620 14:20:26.455394 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.451827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43635 (local address 127.0.98.2:45151)
0620 14:20:26.452166 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.452182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.452316 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.452584 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.452586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.452609 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:26.452763 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.452768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.453501 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.453506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.454291 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.454299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.454933 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.454936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.455035 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.455063 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.455154 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.455227 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":72,"thread_start_us":67,"threads_started":1}
I20260620 14:20:26.456027 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c53e9e53b704c09a7fbde651823e66a" candidate_uuid: "696424886b604f299c62f879ede74dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fe91090fe145fcb38d1dc05f05ec51" is_pre_election: true
I20260620 14:20:26.456066 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c53e9e53b704c09a7fbde651823e66a" candidate_uuid: "696424886b604f299c62f879ede74dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af3cff9f192489296a0ca727e919e43" is_pre_election: true
I20260620 14:20:26.456329 1668 raft_consensus.cc:2454] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696424886b604f299c62f879ede74dd7 in term 0.
I20260620 14:20:26.456360 1947 raft_consensus.cc:2454] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696424886b604f299c62f879ede74dd7 in term 0.
I20260620 14:20:26.456810 1740 leader_election.cc:302] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5af3cff9f192489296a0ca727e919e43, 696424886b604f299c62f879ede74dd7, 83fe91090fe145fcb38d1dc05f05ec51; no voters:
I20260620 14:20:26.457072 2023 raft_consensus.cc:2781] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:26.457154 2023 raft_consensus.cc:482] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:26.457196 2023 raft_consensus.cc:3037] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:26.458191 2023 raft_consensus.cc:504] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260620 14:20:26.458504 2023 leader_election.cc:288] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [CANDIDATE]: Term 1 election: Requested vote from peers 5af3cff9f192489296a0ca727e919e43 (127.0.98.3:43635), 83fe91090fe145fcb38d1dc05f05ec51 (127.0.98.1:45721)
I20260620 14:20:26.458809 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c53e9e53b704c09a7fbde651823e66a" candidate_uuid: "696424886b604f299c62f879ede74dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af3cff9f192489296a0ca727e919e43"
I20260620 14:20:26.458830 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c53e9e53b704c09a7fbde651823e66a" candidate_uuid: "696424886b604f299c62f879ede74dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fe91090fe145fcb38d1dc05f05ec51"
I20260620 14:20:26.458949 1947 raft_consensus.cc:3037] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:26.458966 1668 raft_consensus.cc:3037] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:26.460187 1668 raft_consensus.cc:2454] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696424886b604f299c62f879ede74dd7 in term 1.
I20260620 14:20:26.460325 1947 raft_consensus.cc:2454] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696424886b604f299c62f879ede74dd7 in term 1.
I20260620 14:20:26.460505 1740 leader_election.cc:302] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696424886b604f299c62f879ede74dd7, 83fe91090fe145fcb38d1dc05f05ec51; no voters:
I20260620 14:20:26.460701 2023 raft_consensus.cc:2781] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:26.461035 2023 raft_consensus.cc:686] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 696424886b604f299c62f879ede74dd7, State: Running, Role: LEADER
I20260620 14:20:26.461365 2023 consensus_queue.cc:237] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260620 14:20:26.464427 1511 catalog_manager.cc:5495] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696424886b604f299c62f879ede74dd7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "696424886b604f299c62f879ede74dd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } health_report { overall_health: HEALTHY } } }
I20260620 14:20:26.494511 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.490648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.0.98.2:36877)
0620 14:20:26.490781 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.490785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.490886 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:20:26.491134 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.491136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.491188 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.491293 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:26.491563 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.492157 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.493040 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.493262 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.493369 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.493722 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:26.494113 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:26.494236 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.494314 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.494368 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:20:26.500074 1947 raft_consensus.cc:1261] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [term 1 FOLLOWER]: Refusing update from remote peer 696424886b604f299c62f879ede74dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:26.500098 1668 raft_consensus.cc:1261] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Refusing update from remote peer 696424886b604f299c62f879ede74dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:26.500739 2028 consensus_queue.cc:1035] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:26.500923 2023 consensus_queue.cc:1035] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:26.507984 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298929567737786368 to 7298929567588548608. Current Snapshot: MvccSnapshot[applied={T|T < 7298929567737786368}]
I20260620 14:20:26.509455 2033 mvcc.cc:205] Tried to move back new op lower bound from 7298929567737786368 to 7298929567588548608. Current Snapshot: MvccSnapshot[applied={T|T < 7298929567737786368}]
I20260620 14:20:26.518867 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:26.519061 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:26.521556 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543: Sending DeleteTablet for 3 replicas of tablet 2c53e9e53b704c09a7fbde651823e66a
I20260620 14:20:26.522292 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c53e9e53b704c09a7fbde651823e66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36044
I20260620 14:20:26.522392 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c53e9e53b704c09a7fbde651823e66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46526
I20260620 14:20:26.522516 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c53e9e53b704c09a7fbde651823e66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58148
I20260620 14:20:26.522974 2042 tablet_replica.cc:330] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: stopping tablet replica
I20260620 14:20:26.523010 2040 tablet_replica.cc:330] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: stopping tablet replica
I20260620 14:20:26.523073 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:20:26.523339 2040 raft_consensus.cc:2229] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 14:20:26.523458 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:26.523597 2042 raft_consensus.cc:2229] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:26.523690 2040 raft_consensus.cc:2258] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:26.524060 2042 raft_consensus.cc:2258] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:26.524202 2041 tablet_replica.cc:330] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: stopping tablet replica
I20260620 14:20:26.524475 2041 raft_consensus.cc:2229] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:26.524855 2041 raft_consensus.cc:2258] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:26.525661 2042 ts_tablet_manager.cc:1905] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:26.526458 2040 ts_tablet_manager.cc:1905] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:26.528554 2042 ts_tablet_manager.cc:1918] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:26.528704 2042 log.cc:1196] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/2c53e9e53b704c09a7fbde651823e66a
I20260620 14:20:26.529066 2040 ts_tablet_manager.cc:1918] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:26.529168 2040 log.cc:1196] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/2c53e9e53b704c09a7fbde651823e66a
I20260620 14:20:26.529099 2042 ts_tablet_manager.cc:1939] T 2c53e9e53b704c09a7fbde651823e66a P 696424886b604f299c62f879ede74dd7: Deleting consensus metadata
I20260620 14:20:26.529493 2040 ts_tablet_manager.cc:1939] T 2c53e9e53b704c09a7fbde651823e66a P 5af3cff9f192489296a0ca727e919e43: Deleting consensus metadata
I20260620 14:20:26.530022 1648 tablet_service.cc:1467] Processing CreateTablet for tablet fa9bd2e76dbb4d768a31cabaa517e0c1 (DEFAULT_TABLE table=test-table [id=5d2ad6c393e94661aafbf708708db3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:26.530292 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9bd2e76dbb4d768a31cabaa517e0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:26.530548 1928 tablet_service.cc:1467] Processing CreateTablet for tablet fa9bd2e76dbb4d768a31cabaa517e0c1 (DEFAULT_TABLE table=test-table [id=5d2ad6c393e94661aafbf708708db3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:26.530788 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9bd2e76dbb4d768a31cabaa517e0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:26.532039 1788 tablet_service.cc:1467] Processing CreateTablet for tablet fa9bd2e76dbb4d768a31cabaa517e0c1 (DEFAULT_TABLE table=test-table [id=5d2ad6c393e94661aafbf708708db3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:26.532280 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9bd2e76dbb4d768a31cabaa517e0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:26.532495 2018 tablet_bootstrap.cc:492] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: Bootstrap starting.
I20260620 14:20:26.532557 2019 tablet_bootstrap.cc:492] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: Bootstrap starting.
I20260620 14:20:26.533456 2018 tablet_bootstrap.cc:654] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:26.533461 2019 tablet_bootstrap.cc:654] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:26.534509 2017 tablet_bootstrap.cc:492] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: Bootstrap starting.
I20260620 14:20:26.534642 2019 tablet_bootstrap.cc:492] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: No bootstrap required, opened a new log
I20260620 14:20:26.534708 2019 ts_tablet_manager.cc:1397] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:20:26.534708 1497 catalog_manager.cc:4841] TS 5af3cff9f192489296a0ca727e919e43 (127.0.98.3:43635): tablet 2c53e9e53b704c09a7fbde651823e66a (table test-table [id=e5ba7836e5554c0096518de4d4497f9f]) successfully deleted
I20260620 14:20:26.534860 2018 tablet_bootstrap.cc:492] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: No bootstrap required, opened a new log
I20260620 14:20:26.534931 2018 ts_tablet_manager.cc:1397] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:20:26.535118 1496 catalog_manager.cc:4841] TS 696424886b604f299c62f879ede74dd7 (127.0.98.2:36877): tablet 2c53e9e53b704c09a7fbde651823e66a (table test-table [id=e5ba7836e5554c0096518de4d4497f9f]) successfully deleted
I20260620 14:20:26.535132 2019 raft_consensus.cc:348] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.535233 2019 raft_consensus.cc:374] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:26.535265 2019 raft_consensus.cc:729] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af3cff9f192489296a0ca727e919e43, State: Initialized, Role: FOLLOWER
I20260620 14:20:26.535441 2017 tablet_bootstrap.cc:654] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:26.535403 2019 consensus_queue.cc:260] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.535581 2018 raft_consensus.cc:348] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.535667 2019 ts_tablet_manager.cc:1428] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:20:26.535697 2018 raft_consensus.cc:374] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:26.535732 2018 raft_consensus.cc:729] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fe91090fe145fcb38d1dc05f05ec51, State: Initialized, Role: FOLLOWER
I20260620 14:20:26.535876 2018 consensus_queue.cc:260] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.536113 2018 ts_tablet_manager.cc:1428] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:26.537267 2017 tablet_bootstrap.cc:492] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: No bootstrap required, opened a new log
I20260620 14:20:26.537362 2017 ts_tablet_manager.cc:1397] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 14:20:26.537784 2041 ts_tablet_manager.cc:1905] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:26.537840 2017 raft_consensus.cc:348] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.537971 2017 raft_consensus.cc:374] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:26.538018 2017 raft_consensus.cc:729] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696424886b604f299c62f879ede74dd7, State: Initialized, Role: FOLLOWER
I20260620 14:20:26.538157 2017 consensus_queue.cc:260] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.538460 2017 ts_tablet_manager.cc:1428] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:26.540207 2041 ts_tablet_manager.cc:1918] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:26.540298 2041 log.cc:1196] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/2c53e9e53b704c09a7fbde651823e66a
I20260620 14:20:26.540581 2041 ts_tablet_manager.cc:1939] T 2c53e9e53b704c09a7fbde651823e66a P 83fe91090fe145fcb38d1dc05f05ec51: Deleting consensus metadata
I20260620 14:20:26.541651 1496 catalog_manager.cc:4841] TS 83fe91090fe145fcb38d1dc05f05ec51 (127.0.98.1:45721): tablet 2c53e9e53b704c09a7fbde651823e66a (table test-table [id=e5ba7836e5554c0096518de4d4497f9f]) successfully deleted
I20260620 14:20:26.602367 2024 raft_consensus.cc:482] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:26.602499 2024 raft_consensus.cc:504] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.603665 2024 leader_election.cc:288] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83fe91090fe145fcb38d1dc05f05ec51 (127.0.98.1:45721), 696424886b604f299c62f879ede74dd7 (127.0.98.2:36877)
I20260620 14:20:26.607990 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.603899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45721 (local address 127.0.98.3:32927)
0620 14:20:26.604022 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.604038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.604139 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.604562 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.604567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.604591 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:26.604739 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.604753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.605585 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.605591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.606718 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.606727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.607541 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.607546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.607659 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.607687 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.607753 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.607853 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"spinlock_wait_cycles":2432}
I20260620 14:20:26.608022 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.604018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32927 (local address 127.0.98.1:45721)
0620 14:20:26.604158 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.604163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.604180 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:20:26.604252 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.604255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.604301 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.604381 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:26.604865 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.605468 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.606842 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.607416 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.607534 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.607597 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.607850 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.607917 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:20:26.608520 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9bd2e76dbb4d768a31cabaa517e0c1" candidate_uuid: "5af3cff9f192489296a0ca727e919e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fe91090fe145fcb38d1dc05f05ec51" is_pre_election: true
I20260620 14:20:26.608697 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.604395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35467 (local address 127.0.98.2:36877)
0620 14:20:26.604505 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.604510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.604617 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:20:26.604709 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.604712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.604755 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.604824 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:26.605125 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.605881 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.607150 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.607586 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.607687 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.607760 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.608503 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.608585 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:20:26.608686 1668 raft_consensus.cc:2454] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5af3cff9f192489296a0ca727e919e43 in term 0.
I20260620 14:20:26.608879 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.604161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36877 (local address 127.0.98.3:35467)
0620 14:20:26.604504 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.604517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.604629 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.604883 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.604886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.604903 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:26.605012 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.605018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.606002 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.606005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.607030 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.607037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.607707 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.607711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.608353 (+ 642us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.608379 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.608469 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.608534 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":85,"threads_started":1}
I20260620 14:20:26.609090 1880 leader_election.cc:302] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af3cff9f192489296a0ca727e919e43, 83fe91090fe145fcb38d1dc05f05ec51; no voters:
I20260620 14:20:26.609324 2024 raft_consensus.cc:2781] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:26.609386 2024 raft_consensus.cc:482] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:26.609429 2024 raft_consensus.cc:3037] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:26.609761 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9bd2e76dbb4d768a31cabaa517e0c1" candidate_uuid: "5af3cff9f192489296a0ca727e919e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696424886b604f299c62f879ede74dd7" is_pre_election: true
I20260620 14:20:26.609972 1808 raft_consensus.cc:2454] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5af3cff9f192489296a0ca727e919e43 in term 0.
I20260620 14:20:26.610687 2024 raft_consensus.cc:504] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.611047 2024 leader_election.cc:288] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [CANDIDATE]: Term 1 election: Requested vote from peers 83fe91090fe145fcb38d1dc05f05ec51 (127.0.98.1:45721), 696424886b604f299c62f879ede74dd7 (127.0.98.2:36877)
I20260620 14:20:26.611371 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9bd2e76dbb4d768a31cabaa517e0c1" candidate_uuid: "5af3cff9f192489296a0ca727e919e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fe91090fe145fcb38d1dc05f05ec51"
I20260620 14:20:26.611385 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9bd2e76dbb4d768a31cabaa517e0c1" candidate_uuid: "5af3cff9f192489296a0ca727e919e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696424886b604f299c62f879ede74dd7"
I20260620 14:20:26.611508 1808 raft_consensus.cc:3037] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:26.611514 1668 raft_consensus.cc:3037] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:26.612463 1668 raft_consensus.cc:2454] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af3cff9f192489296a0ca727e919e43 in term 1.
I20260620 14:20:26.612682 1808 raft_consensus.cc:2454] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af3cff9f192489296a0ca727e919e43 in term 1.
I20260620 14:20:26.612730 1880 leader_election.cc:302] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af3cff9f192489296a0ca727e919e43, 83fe91090fe145fcb38d1dc05f05ec51; no voters:
I20260620 14:20:26.612900 2024 raft_consensus.cc:2781] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:26.613090 2024 raft_consensus.cc:686] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 1 LEADER]: Becoming Leader. State: Replica: 5af3cff9f192489296a0ca727e919e43, State: Running, Role: LEADER
I20260620 14:20:26.613320 2024 consensus_queue.cc:237] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.615562 1511 catalog_manager.cc:5495] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5af3cff9f192489296a0ca727e919e43 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5af3cff9f192489296a0ca727e919e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } health_report { overall_health: HEALTHY } } }
Jun 20 14:20:26 dist-test-slave-87nc krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:26.666997 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.657695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.0.98.62:46049)
0620 14:20:26.657815 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.657820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.658017 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:20:26.658195 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.658197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.658238 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.658315 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:26.659162 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.659688 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.660976 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.661200 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.661286 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.663728 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:26.663737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:26.663739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:26.663757 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:26.665299 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.665812 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.665815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.665816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.665886 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.666154 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.666156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.666158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.666310 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:26.666381 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.666694 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.666875 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":49}
I20260620 14:20:26.667004 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.657565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46049 (local address 127.0.0.1:56876)
0620 14:20:26.657916 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.657933 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.658057 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.658371 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.658374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.658916 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:26.659033 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.659042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.659820 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.659824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.660751 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.660762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.661321 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.661325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.661421 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.662000 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:26.662014 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:26.663597 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:26.665418 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:26.665427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:26.665433 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:26.665708 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:26.665971 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:26.665975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:26.665979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:26.666069 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:26.666409 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:26.666417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:26.666535 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.666648 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.666805 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":128,"threads_started":1}
I20260620 14:20:26.671582 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56876:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:20:26.672379 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:26.677269 1788 tablet_service.cc:1467] Processing CreateTablet for tablet a123ab985dfa464aa2155c20252d3356 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dc4bb4579bb40ecb4b599049ae3c827]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:20:26.677265 1648 tablet_service.cc:1467] Processing CreateTablet for tablet a123ab985dfa464aa2155c20252d3356 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dc4bb4579bb40ecb4b599049ae3c827]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:20:26.677641 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a123ab985dfa464aa2155c20252d3356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:26.677641 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a123ab985dfa464aa2155c20252d3356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:26.680284 2017 tablet_bootstrap.cc:492] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: Bootstrap starting.
I20260620 14:20:26.680191 1928 tablet_service.cc:1467] Processing CreateTablet for tablet a123ab985dfa464aa2155c20252d3356 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dc4bb4579bb40ecb4b599049ae3c827]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:20:26.680523 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a123ab985dfa464aa2155c20252d3356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:26.681010 2017 tablet_bootstrap.cc:654] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:26.681579 2018 tablet_bootstrap.cc:492] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: Bootstrap starting.
I20260620 14:20:26.682307 2018 tablet_bootstrap.cc:654] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:26.683221 2017 tablet_bootstrap.cc:492] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: No bootstrap required, opened a new log
I20260620 14:20:26.683303 2017 ts_tablet_manager.cc:1397] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:20:26.683728 2019 tablet_bootstrap.cc:492] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: Bootstrap starting.
I20260620 14:20:26.683722 2017 raft_consensus.cc:348] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.683898 2017 raft_consensus.cc:374] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:26.683944 2017 raft_consensus.cc:729] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696424886b604f299c62f879ede74dd7, State: Initialized, Role: FOLLOWER
I20260620 14:20:26.684087 2017 consensus_queue.cc:260] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.684355 2017 ts_tablet_manager.cc:1428] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:26.684582 2023 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.684708 2023 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:26.685045 2019 tablet_bootstrap.cc:654] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:26.685516 2018 tablet_bootstrap.cc:492] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: No bootstrap required, opened a new log
I20260620 14:20:26.685606 2018 ts_tablet_manager.cc:1397] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:20:26.686065 2018 raft_consensus.cc:348] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.686183 2018 raft_consensus.cc:374] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:26.686224 2018 raft_consensus.cc:729] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fe91090fe145fcb38d1dc05f05ec51, State: Initialized, Role: FOLLOWER
I20260620 14:20:26.686354 2018 consensus_queue.cc:260] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.686610 2018 ts_tablet_manager.cc:1428] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:26.686774 2019 tablet_bootstrap.cc:492] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: No bootstrap required, opened a new log
I20260620 14:20:26.686854 2019 ts_tablet_manager.cc:1397] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:20:26.686800 2025 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.686949 2025 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:26.687219 2019 raft_consensus.cc:348] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.687320 2019 raft_consensus.cc:374] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:26.687346 2019 raft_consensus.cc:729] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af3cff9f192489296a0ca727e919e43, State: Initialized, Role: FOLLOWER
I20260620 14:20:26.687460 2019 consensus_queue.cc:260] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.688505 2019 ts_tablet_manager.cc:1428] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:26.688459 2024 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.688618 2024 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:26.804312 2024 raft_consensus.cc:482] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:26.804481 2024 raft_consensus.cc:504] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.804903 2024 leader_election.cc:288] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83fe91090fe145fcb38d1dc05f05ec51 (127.0.98.1:45721), 696424886b604f299c62f879ede74dd7 (127.0.98.2:36877)
I20260620 14:20:26.805366 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a123ab985dfa464aa2155c20252d3356" candidate_uuid: "5af3cff9f192489296a0ca727e919e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696424886b604f299c62f879ede74dd7" is_pre_election: true
I20260620 14:20:26.805382 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a123ab985dfa464aa2155c20252d3356" candidate_uuid: "5af3cff9f192489296a0ca727e919e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fe91090fe145fcb38d1dc05f05ec51" is_pre_election: true
I20260620 14:20:26.805528 1808 raft_consensus.cc:2454] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5af3cff9f192489296a0ca727e919e43 in term 0.
I20260620 14:20:26.805549 1668 raft_consensus.cc:2454] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5af3cff9f192489296a0ca727e919e43 in term 0.
I20260620 14:20:26.805815 1880 leader_election.cc:302] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af3cff9f192489296a0ca727e919e43, 83fe91090fe145fcb38d1dc05f05ec51; no voters:
I20260620 14:20:26.806025 2024 raft_consensus.cc:2781] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:26.806106 2024 raft_consensus.cc:482] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:26.806162 2024 raft_consensus.cc:3037] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:26.807106 2024 raft_consensus.cc:504] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.807420 2024 leader_election.cc:288] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [CANDIDATE]: Term 1 election: Requested vote from peers 83fe91090fe145fcb38d1dc05f05ec51 (127.0.98.1:45721), 696424886b604f299c62f879ede74dd7 (127.0.98.2:36877)
I20260620 14:20:26.807813 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a123ab985dfa464aa2155c20252d3356" candidate_uuid: "5af3cff9f192489296a0ca727e919e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fe91090fe145fcb38d1dc05f05ec51"
I20260620 14:20:26.807842 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a123ab985dfa464aa2155c20252d3356" candidate_uuid: "5af3cff9f192489296a0ca727e919e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696424886b604f299c62f879ede74dd7"
I20260620 14:20:26.807930 1668 raft_consensus.cc:3037] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:26.807963 1808 raft_consensus.cc:3037] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:26.808811 1668 raft_consensus.cc:2454] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af3cff9f192489296a0ca727e919e43 in term 1.
I20260620 14:20:26.808827 1808 raft_consensus.cc:2454] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af3cff9f192489296a0ca727e919e43 in term 1.
I20260620 14:20:26.809077 1880 leader_election.cc:302] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af3cff9f192489296a0ca727e919e43, 83fe91090fe145fcb38d1dc05f05ec51; no voters:
I20260620 14:20:26.809233 2024 raft_consensus.cc:2781] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:26.809361 2024 raft_consensus.cc:686] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [term 1 LEADER]: Becoming Leader. State: Replica: 5af3cff9f192489296a0ca727e919e43, State: Running, Role: LEADER
I20260620 14:20:26.809505 2024 consensus_queue.cc:237] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } }
I20260620 14:20:26.810118 2047 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: TxnStatusTablet state changed. Reason: New leader 5af3cff9f192489296a0ca727e919e43. Latest consensus state: current_term: 1 leader_uuid: "5af3cff9f192489296a0ca727e919e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.810228 2047 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:20:26.810536 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:20:26.810702 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:20:26.811126 1510 catalog_manager.cc:5495] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5af3cff9f192489296a0ca727e919e43 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5af3cff9f192489296a0ca727e919e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } health_report { overall_health: HEALTHY } } }
I20260620 14:20:26.844341 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.840421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.0.98.3:43635)
0620 14:20:26.840559 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.840564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.840580 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:20:26.840655 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.840659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.840715 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.840816 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:26.841047 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.841543 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.842855 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.843150 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.843307 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.843595 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:26.843996 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:26.844081 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.844192 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.844236 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:20:26.844422 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.840296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43635 (local address 127.0.0.1:39298)
0620 14:20:26.840414 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.840431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.840529 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.840824 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.840827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.840835 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:26.840937 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.840943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.841708 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.841712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.842748 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.842760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.843314 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.843318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.843410 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.843461 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:26.844088 (+ 627us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:26.844103 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.844183 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.844271 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:20:26.847070 1668 raft_consensus.cc:1261] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Refusing update from remote peer 5af3cff9f192489296a0ca727e919e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:26.847235 1808 raft_consensus.cc:1261] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7 [term 1 FOLLOWER]: Refusing update from remote peer 5af3cff9f192489296a0ca727e919e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:26.847564 2024 consensus_queue.cc:1035] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:26.847803 2047 consensus_queue.cc:1035] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:26.849103 2025 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: TxnStatusTablet state changed. Reason: New leader 5af3cff9f192489296a0ca727e919e43. Latest consensus state: current_term: 1 leader_uuid: "5af3cff9f192489296a0ca727e919e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.849212 2025 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:26.849943 2023 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: TxnStatusTablet state changed. Reason: New leader 5af3cff9f192489296a0ca727e919e43. Latest consensus state: current_term: 1 leader_uuid: "5af3cff9f192489296a0ca727e919e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.850064 2023 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:26.852238 2024 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5af3cff9f192489296a0ca727e919e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.852342 2024 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:20:26.852468 2024 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5af3cff9f192489296a0ca727e919e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.852545 2024 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 5af3cff9f192489296a0ca727e919e43: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:20:26.852507 2025 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5af3cff9f192489296a0ca727e919e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.852596 2025 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 83fe91090fe145fcb38d1dc05f05ec51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:26.855145 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298929569160675328 to 7298929569013039104. Current Snapshot: MvccSnapshot[applied={T|T < 7298929569160675328 or (T in {7298929569160675328})}]
I20260620 14:20:26.855342 2023 tablet_replica.cc:439] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5af3cff9f192489296a0ca727e919e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "5af3cff9f192489296a0ca727e919e43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } }
I20260620 14:20:26.855437 2023 tablet_replica.cc:442] T a123ab985dfa464aa2155c20252d3356 P 696424886b604f299c62f879ede74dd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:26.860169 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.857250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.0.98.3:43635)
0620 14:20:26.857343 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.857348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.857384 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:20:26.857535 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.857539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.857596 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.857674 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:26.857894 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.858377 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.859173 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.859399 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.859504 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.859735 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:26.859844 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:26.859921 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.860006 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.860049 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:20:26.870167 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.866671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59827 (local address 127.0.98.3:43635)
0620 14:20:26.866767 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.866770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.866983 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:20:26.867085 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.867087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.867121 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.867180 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:26.867456 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.867972 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.869190 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.869379 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.869451 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.869771 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:26.869844 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:26.869919 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.870026 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.870068 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:20:26.870283 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.866597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43635 (local address 127.0.98.3:59827)
0620 14:20:26.866895 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:26.866909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:26.867013 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:26.867215 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:26.867218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:26.867226 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:26.867361 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.867367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.868065 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.868068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.869078 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:26.869088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.869481 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.869485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:26.869577 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.869635 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:26.869932 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:26.869938 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:20:26.870003 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:20:26.870074 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260620 14:20:26.876084 1668 raft_consensus.cc:1261] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Refusing update from remote peer 5af3cff9f192489296a0ca727e919e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:26.876087 1808 raft_consensus.cc:1261] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [term 1 FOLLOWER]: Refusing update from remote peer 5af3cff9f192489296a0ca727e919e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:26.876488 2064 consensus_queue.cc:1035] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fe91090fe145fcb38d1dc05f05ec51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:26.876645 2047 consensus_queue.cc:1035] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696424886b604f299c62f879ede74dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:26.918990 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:26.919173 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:26.921605 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0b111c3b5b4631ba8c0c919380a543: Sending DeleteTablet for 3 replicas of tablet fa9bd2e76dbb4d768a31cabaa517e0c1
I20260620 14:20:26.922227 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9bd2e76dbb4d768a31cabaa517e0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46526
I20260620 14:20:26.922340 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9bd2e76dbb4d768a31cabaa517e0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36044
I20260620 14:20:26.922513 2040 tablet_replica.cc:330] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: stopping tablet replica
I20260620 14:20:26.922535 2041 tablet_replica.cc:330] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: stopping tablet replica
I20260620 14:20:26.922626 2040 raft_consensus.cc:2229] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:26.922650 2041 raft_consensus.cc:2229] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:26.922708 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9bd2e76dbb4d768a31cabaa517e0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58148
I20260620 14:20:26.922762 2041 raft_consensus.cc:2258] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:26.922914 2042 tablet_replica.cc:330] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: stopping tablet replica
I20260620 14:20:26.922915 2040 raft_consensus.cc:2258] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:26.923060 2042 raft_consensus.cc:2229] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:26.923187 2042 raft_consensus.cc:2258] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:26.924031 2040 ts_tablet_manager.cc:1905] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:26.924031 2042 ts_tablet_manager.cc:1905] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:26.926867 2042 ts_tablet_manager.cc:1918] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:26.926872 2040 ts_tablet_manager.cc:1918] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:26.926944 2040 log.cc:1196] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/fa9bd2e76dbb4d768a31cabaa517e0c1
I20260620 14:20:26.926944 2042 log.cc:1196] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/fa9bd2e76dbb4d768a31cabaa517e0c1
I20260620 14:20:26.927223 2042 ts_tablet_manager.cc:1939] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 696424886b604f299c62f879ede74dd7: Deleting consensus metadata
I20260620 14:20:26.927222 2040 ts_tablet_manager.cc:1939] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 5af3cff9f192489296a0ca727e919e43: Deleting consensus metadata
I20260620 14:20:26.927930 1496 catalog_manager.cc:4841] TS 696424886b604f299c62f879ede74dd7 (127.0.98.2:36877): tablet fa9bd2e76dbb4d768a31cabaa517e0c1 (table test-table [id=5d2ad6c393e94661aafbf708708db3a0]) successfully deleted
I20260620 14:20:26.927974 1497 catalog_manager.cc:4841] TS 5af3cff9f192489296a0ca727e919e43 (127.0.98.3:43635): tablet fa9bd2e76dbb4d768a31cabaa517e0c1 (table test-table [id=5d2ad6c393e94661aafbf708708db3a0]) successfully deleted
I20260620 14:20:26.928781 2041 ts_tablet_manager.cc:1905] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:26.931552 2041 ts_tablet_manager.cc:1918] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:26.931658 2041 log.cc:1196] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/fa9bd2e76dbb4d768a31cabaa517e0c1
I20260620 14:20:26.932343 2041 ts_tablet_manager.cc:1939] T fa9bd2e76dbb4d768a31cabaa517e0c1 P 83fe91090fe145fcb38d1dc05f05ec51: Deleting consensus metadata
I20260620 14:20:26.933082 1496 catalog_manager.cc:4841] TS 83fe91090fe145fcb38d1dc05f05ec51 (127.0.98.1:45721): tablet fa9bd2e76dbb4d768a31cabaa517e0c1 (table test-table [id=5d2ad6c393e94661aafbf708708db3a0]) successfully deleted
Jun 20 14:20:26 dist-test-slave-87nc krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965226, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:26.945093 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.932977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.0.98.1:45721)
0620 14:20:26.933079 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.933084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.933099 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:26.933138 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.933141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.933188 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.933270 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:26.934273 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.935072 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.936304 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.936665 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.936770 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.940919 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:26.940930 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:26.940946 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:26.940984 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:26.943285 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.943734 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.943741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.943743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.943863 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.944128 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.944132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.944133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.944427 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:26.944626 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.944790 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.944920 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 14:20:26.945668 1708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44414
I20260620 14:20:26.955237 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:26.947466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.0.98.62:46049)
0620 14:20:26.947597 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:26.947603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:26.947621 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:20:26.947702 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:26.947704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:26.947743 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:26.947863 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:26.948631 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.949165 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.950247 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:26.950457 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:26.950529 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:26.952279 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:26.952282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:26.952284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:26.952306 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:26.953680 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.954141 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.954143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.954144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.954211 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:26.954485 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:26.954487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:26.954488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:26.954641 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:26.954735 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:26.954991 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:20:26.955086 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 14:20:26.955657 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56878
I20260620 14:20:26.957083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 1583
I20260620 14:20:26.964993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 1723
I20260620 14:20:26.974399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 1863
I20260620 14:20:26.981237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 1479
2026-06-20T14:20:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4211 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:27 dist-test-slave-87nc krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:27 dist-test-slave-87nc krb5kdc[2077](info): set up 2 sockets
Jun 20 14:20:27 dist-test-slave-87nc krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260620 14:20:28.023689 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:28 dist-test-slave-87nc krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:28.140836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45371
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:28.251040 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:28.251379 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:28.251457 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:28.256001 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:28.256101 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:20:28.256131 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:28.256177 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:28.256215 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:28.256248 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:28.261686 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45371
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:28.262773 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:28.263860 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:28.272341 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:28.272336 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:28.272457 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:28.272583 2090 server_base.cc:1034] running on GCE node
I20260620 14:20:28.273186 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:28.274380 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:28.275574 2090 hybrid_clock.cc:648] HybridClock initialized: now 1781965228275551 us; error 34 us; skew 500 ppm
Jun 20 14:20:28 dist-test-slave-87nc krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:28.278381 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:28.279270 2090 webserver.cc:460] Webserver started at http://127.0.98.62:41669/ using document root <none> and password file <none>
I20260620 14:20:28.279924 2090 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:28.280015 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:28.280228 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:28.282238 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "292985ee8b834703996a043366dd992f"
format_stamp: "Formatted at 2026-06-20 14:20:28 on dist-test-slave-87nc"
server_key: "5e2982aae44878099d0d0c09a841b482"
server_key_iv: "597cc448f2ef18d65239614ef9a56cf6"
server_key_version: "encryptionkey@0"
I20260620 14:20:28.282793 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "292985ee8b834703996a043366dd992f"
format_stamp: "Formatted at 2026-06-20 14:20:28 on dist-test-slave-87nc"
server_key: "5e2982aae44878099d0d0c09a841b482"
server_key_iv: "597cc448f2ef18d65239614ef9a56cf6"
server_key_version: "encryptionkey@0"
I20260620 14:20:28.286526 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:20:28.289202 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:28.290458 2090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:28.290603 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "292985ee8b834703996a043366dd992f"
format_stamp: "Formatted at 2026-06-20 14:20:28 on dist-test-slave-87nc"
server_key: "5e2982aae44878099d0d0c09a841b482"
server_key_iv: "597cc448f2ef18d65239614ef9a56cf6"
server_key_version: "encryptionkey@0"
I20260620 14:20:28.290719 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:28.313009 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:28.316468 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:28.316679 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:28.324479 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45371
I20260620 14:20:28.324502 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45371 every 8 connection(s)
I20260620 14:20:28.325418 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:28.327054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 2090
I20260620 14:20:28.327183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:28.327448 392 external_mini_cluster.cc:1428] Setting key 7403a880ce625223b7272623826b9ea8
I20260620 14:20:28.328375 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:28.333782 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f: Bootstrap starting.
I20260620 14:20:28.336006 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f: Neither blocks nor log segments found. Creating new log.
Jun 20 14:20:28 dist-test-slave-87nc krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:28.336719 2168 log.cc:824] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:28.338527 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f: No bootstrap required, opened a new log
I20260620 14:20:28.341527 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292985ee8b834703996a043366dd992f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } }
I20260620 14:20:28.341732 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:28.341738 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:28.328826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.0.98.62:45371)
0620 14:20:28.329254 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:28.329268 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:28.329305 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:20:28.329982 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:28.329998 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:28.330320 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:28.330678 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:28.331433 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.332516 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.333401 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.333922 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.334081 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:28.336674 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:28.336679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:28.336692 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:28.336719 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:28.339076 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.339506 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.339518 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.339522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.339628 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.339934 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.339939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.339941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.340308 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:28.340523 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:28.340747 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:20:28.341016 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":141,"threads_started":1}
I20260620 14:20:28.341814 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 292985ee8b834703996a043366dd992f, State: Initialized, Role: FOLLOWER
I20260620 14:20:28.342265 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292985ee8b834703996a043366dd992f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } }
I20260620 14:20:28.342423 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:28.342500 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:28.342614 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:28.343475 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292985ee8b834703996a043366dd992f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } }
I20260620 14:20:28.343891 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 292985ee8b834703996a043366dd992f; no voters:
I20260620 14:20:28.344223 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:28.344333 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:28.344530 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [term 1 LEADER]: Becoming Leader. State: Replica: 292985ee8b834703996a043366dd992f, State: Running, Role: LEADER
I20260620 14:20:28.344862 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292985ee8b834703996a043366dd992f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } }
I20260620 14:20:28.345474 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:28.346513 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "292985ee8b834703996a043366dd992f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292985ee8b834703996a043366dd992f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } } }
I20260620 14:20:28.346653 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:28.347498 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 292985ee8b834703996a043366dd992f. Latest consensus state: current_term: 1 leader_uuid: "292985ee8b834703996a043366dd992f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292985ee8b834703996a043366dd992f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } } }
I20260620 14:20:28.347667 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:28.347863 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:28.350771 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:28.356564 2182 catalog_manager.cc:1347] Generated new cluster ID: 00eaeca5bea9496ea8c9e4cf204d5444
I20260620 14:20:28.356690 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:28.373893 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:28.374836 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:28.382730 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f: Generated new TSK 0
I20260620 14:20:28.383489 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:20:28.422246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45371
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:28.535791 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:28.536106 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:28.536198 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:28.540244 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:28.540342 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:28.540447 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:20:28.540496 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:20:28.540575 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:28.545929 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:28.547050 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:28.548207 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:28.558017 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:28.557979 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:28.558040 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:28.558164 2194 server_base.cc:1034] running on GCE node
I20260620 14:20:28.558573 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:28.559223 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:28.560380 2194 hybrid_clock.cc:648] HybridClock initialized: now 1781965228560359 us; error 31 us; skew 500 ppm
Jun 20 14:20:28 dist-test-slave-87nc krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:28.563024 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:28.563848 2194 webserver.cc:460] Webserver started at http://127.0.98.1:38311/ using document root <none> and password file <none>
I20260620 14:20:28.564347 2194 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:28.564404 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:28.564566 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:28.566444 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "a58b25134e6448b4b6b26eed83bc2bbe"
format_stamp: "Formatted at 2026-06-20 14:20:28 on dist-test-slave-87nc"
server_key: "94e454de1abdad40d8395642ec07d12a"
server_key_iv: "1c435fa48f497ed9ef6e34672df27e83"
server_key_version: "encryptionkey@0"
I20260620 14:20:28.566936 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a58b25134e6448b4b6b26eed83bc2bbe"
format_stamp: "Formatted at 2026-06-20 14:20:28 on dist-test-slave-87nc"
server_key: "94e454de1abdad40d8395642ec07d12a"
server_key_iv: "1c435fa48f497ed9ef6e34672df27e83"
server_key_version: "encryptionkey@0"
I20260620 14:20:28.570425 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:20:28.572718 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:28.574079 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:28.574184 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "a58b25134e6448b4b6b26eed83bc2bbe"
format_stamp: "Formatted at 2026-06-20 14:20:28 on dist-test-slave-87nc"
server_key: "94e454de1abdad40d8395642ec07d12a"
server_key_iv: "1c435fa48f497ed9ef6e34672df27e83"
server_key_version: "encryptionkey@0"
I20260620 14:20:28.574267 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:28.592583 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:28.595733 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:28.595898 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:28.597183 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:28.597234 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:28.597296 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:28.597335 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:20:28 dist-test-slave-87nc krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:28.608119 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32787
I20260620 14:20:28.608192 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32787 every 8 connection(s)
I20260620 14:20:28.609259 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:28.611083 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:28.598013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60689 (local address 127.0.98.62:45371)
0620 14:20:28.598175 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:28.598182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:28.598985 (+ 803us) server_negotiation.cc:407] Connection header received
0620 14:20:28.600216 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:28.600223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:28.600296 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:28.600404 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:28.601875 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.602791 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.604231 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.604683 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.604813 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:28.607327 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:28.607331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:28.607334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:28.607364 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:28.609108 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.609643 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.609649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.609654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.609751 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.610148 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.610150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.610151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.610329 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:28.610410 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:28.610850 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 14:20:28.610950 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:20:28.611918 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:28.598368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45371 (local address 127.0.98.1:60689)
0620 14:20:28.598823 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:28.598874 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:28.599906 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:28.600602 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:28.600608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:28.601022 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:28.601654 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:28.601670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.602946 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.602950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:28.604130 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:28.604139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.604859 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.604866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:28.605065 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:28.605651 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:28.605667 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:28.607162 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:28.609215 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:28.609219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:28.609231 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:28.609540 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:28.609857 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:28.609860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:28.609862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:28.610031 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:28.610431 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:28.610436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:28.610655 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 14:20:28.610804 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 14:20:28.611052 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
I20260620 14:20:28.613123 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45371
I20260620 14:20:28.613258 2325 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:28.613806 2325 heartbeater.cc:511] Master 127.0.98.62:45371 requested a full tablet report, sending...
I20260620 14:20:28.615320 2121 ts_manager.cc:194] Registered new tserver with Master: a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787)
I20260620 14:20:28.616503 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60689
I20260620 14:20:28.618800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 2194
I20260620 14:20:28.618944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:28.619207 392 external_mini_cluster.cc:1428] Setting key bece7ef43097876af2137c68c62dfb00
I20260620 14:20:28.622529 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:28.615302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45371 (local address 127.0.98.1:46757)
0620 14:20:28.615632 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:28.615644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:28.615753 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:28.616015 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:28.616017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:28.616214 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:28.616305 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:28.616310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.617178 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.617182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:28.617758 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:28.617765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.618206 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.618210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:28.618302 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:28.618823 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:28.618834 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:28.619258 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:28.621107 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:28.621109 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:28.621110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:28.621289 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:28.621578 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:28.621580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:28.621581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:28.621650 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:28.622069 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:28.622071 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:28.622142 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 14:20:28.622218 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:20:28.622319 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":77,"threads_started":1}
I20260620 14:20:28.622871 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:28.615412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46757 (local address 127.0.98.62:45371)
0620 14:20:28.615543 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:28.615547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:28.615713 (+ 166us) server_negotiation.cc:407] Connection header received
0620 14:20:28.615863 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:28.615868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:28.615916 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:28.616000 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:28.616418 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.617048 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.617873 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.618095 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.618203 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:28.619409 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:28.619413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:28.619416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:28.619440 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:28.621003 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.621384 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.621390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.621395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.621484 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.621736 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.621743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.621747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.621976 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:28.622086 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:28.622298 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:20:28.622423 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:20:28.647497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45371
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:28.758591 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:28.758839 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:28.758886 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:28.763056 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:28.763114 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:28.763154 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:20:28.763170 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:20:28.763200 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:28.768523 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:28.769554 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:28.770604 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:28.780432 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:28.780546 2334 server_base.cc:1034] running on GCE node
W20260620 14:20:28.780426 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:28.780481 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:28.780984 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:28.781622 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:28.782806 2334 hybrid_clock.cc:648] HybridClock initialized: now 1781965228782766 us; error 52 us; skew 500 ppm
Jun 20 14:20:28 dist-test-slave-87nc krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:28.785501 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:28.786315 2334 webserver.cc:460] Webserver started at http://127.0.98.2:39583/ using document root <none> and password file <none>
I20260620 14:20:28.786871 2334 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:28.786959 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:28.787189 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:28.789366 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "a325f95b734d4a128a4ab18d88c140ce"
format_stamp: "Formatted at 2026-06-20 14:20:28 on dist-test-slave-87nc"
server_key: "a3161cbf623efffcdb8c9a5f09cc353b"
server_key_iv: "56733c0bb77deb88276c5b178aaa19a2"
server_key_version: "encryptionkey@0"
I20260620 14:20:28.789922 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a325f95b734d4a128a4ab18d88c140ce"
format_stamp: "Formatted at 2026-06-20 14:20:28 on dist-test-slave-87nc"
server_key: "a3161cbf623efffcdb8c9a5f09cc353b"
server_key_iv: "56733c0bb77deb88276c5b178aaa19a2"
server_key_version: "encryptionkey@0"
I20260620 14:20:28.793659 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:20:28.796154 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:28.797365 2334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:28.797502 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "a325f95b734d4a128a4ab18d88c140ce"
format_stamp: "Formatted at 2026-06-20 14:20:28 on dist-test-slave-87nc"
server_key: "a3161cbf623efffcdb8c9a5f09cc353b"
server_key_iv: "56733c0bb77deb88276c5b178aaa19a2"
server_key_version: "encryptionkey@0"
I20260620 14:20:28.797617 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:28.821820 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:28.825287 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:28.825510 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:28.826954 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:28.827025 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:28.827128 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:28.827180 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:20:28 dist-test-slave-87nc krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:28.838021 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43971
I20260620 14:20:28.838119 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43971 every 8 connection(s)
I20260620 14:20:28.839097 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:28.841959 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:28.827715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53143 (local address 127.0.98.62:45371)
0620 14:20:28.827881 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:28.827886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:28.828818 (+ 932us) server_negotiation.cc:407] Connection header received
0620 14:20:28.830101 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:28.830108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:28.830177 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:28.830240 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:28.832184 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.832895 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.834801 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.835105 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.835193 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:28.838277 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:28.838281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:28.838284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:28.838321 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:28.839881 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.840385 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.840388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.840391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.840473 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.840821 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.840825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.840826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.841063 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:28.841162 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:28.841668 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 14:20:28.841789 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:20:28.842716 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:28.828145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45371 (local address 127.0.98.2:53143)
0620 14:20:28.828640 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:28.828696 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:28.829830 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:28.830709 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:28.830722 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:28.831196 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:28.831987 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:28.832010 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.833040 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.833044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:28.834647 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:28.834662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.835264 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.835268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:28.835490 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:28.836233 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:28.836258 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:28.838107 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:28.839984 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:28.839989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:28.840004 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:28.840273 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:28.840568 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:28.840572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:28.840574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:28.840719 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:28.841158 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:28.841165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:28.841415 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 14:20:28.841599 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 14:20:28.841922 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20260620 14:20:28.843875 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45371
I20260620 14:20:28.844035 2465 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:28.844310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 2334
I20260620 14:20:28.844406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:28.844622 2465 heartbeater.cc:511] Master 127.0.98.62:45371 requested a full tablet report, sending...
I20260620 14:20:28.844733 392 external_mini_cluster.cc:1428] Setting key 893c36954814d5d6f1a6b07523e61f11
I20260620 14:20:28.845865 2121 ts_manager.cc:194] Registered new tserver with Master: a325f95b734d4a128a4ab18d88c140ce (127.0.98.2:43971)
I20260620 14:20:28.846589 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53143
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:20:28.856144 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:28.846361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45371 (local address 127.0.98.2:51477)
0620 14:20:28.848497 (+ 2136us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:28.848516 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:28.848636 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:28.848894 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:28.848898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:28.849134 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:28.849325 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:28.849334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.850396 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.850398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:28.850960 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:28.850968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.851576 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.851592 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:28.851663 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:28.852278 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:28.852290 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:28.852640 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:28.854548 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:28.854550 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:28.854552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:28.854759 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:28.855053 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:28.855055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:28.855056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:28.855127 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:28.855610 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:28.855612 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:28.855741 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 14:20:28.855871 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:20:28.855989 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2039,"thread_start_us":111,"threads_started":1}
I20260620 14:20:28.856194 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:28.846908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51477 (local address 127.0.98.62:45371)
0620 14:20:28.846999 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:28.847003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:28.848600 (+ 1597us) server_negotiation.cc:407] Connection header received
0620 14:20:28.848749 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:28.848751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:28.848787 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:28.848852 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:28.849443 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.850273 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.851081 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:28.851453 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:28.851594 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:28.852782 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:28.852789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:28.852794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:28.852825 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:28.854439 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.854860 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.854865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.854868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.854955 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:28.855218 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:28.855222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:28.855224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:28.855512 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:28.855625 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:28.855939 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 14:20:28.856073 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:20:28.874657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45371
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:28.984932 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:28.985222 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:28.985271 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:28.989367 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:28.989432 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:28.989511 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:20:28.989553 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:20:28.989635 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:28.995268 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:28.996371 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:28.997392 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:29.007002 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:29.007141 2474 server_base.cc:1034] running on GCE node
W20260620 14:20:29.007010 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:29.007025 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:29.007849 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:29.008410 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:29.009580 2474 hybrid_clock.cc:648] HybridClock initialized: now 1781965229009574 us; error 43 us; skew 500 ppm
Jun 20 14:20:29 dist-test-slave-87nc krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:29.012231 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:29.013104 2474 webserver.cc:460] Webserver started at http://127.0.98.3:33845/ using document root <none> and password file <none>
I20260620 14:20:29.013629 2474 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:29.013686 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:29.013916 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:29.015913 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "4db0ca64ef454dceacd54070fb218bd0"
format_stamp: "Formatted at 2026-06-20 14:20:29 on dist-test-slave-87nc"
server_key: "8cd1bee045b5d22efc2e4e5cd5547333"
server_key_iv: "7ac935f117693153a12965d33a2f97a9"
server_key_version: "encryptionkey@0"
I20260620 14:20:29.016448 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4db0ca64ef454dceacd54070fb218bd0"
format_stamp: "Formatted at 2026-06-20 14:20:29 on dist-test-slave-87nc"
server_key: "8cd1bee045b5d22efc2e4e5cd5547333"
server_key_iv: "7ac935f117693153a12965d33a2f97a9"
server_key_version: "encryptionkey@0"
I20260620 14:20:29.019955 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:20:29.022151 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:29.023300 2474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:29.023411 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "4db0ca64ef454dceacd54070fb218bd0"
format_stamp: "Formatted at 2026-06-20 14:20:29 on dist-test-slave-87nc"
server_key: "8cd1bee045b5d22efc2e4e5cd5547333"
server_key_iv: "7ac935f117693153a12965d33a2f97a9"
server_key_version: "encryptionkey@0"
I20260620 14:20:29.023502 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:29.045518 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:29.048599 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:29.048797 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:29.050215 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:29.050289 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:29.050366 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:29.050410 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:20:29 dist-test-slave-87nc krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:29.061196 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33679
I20260620 14:20:29.061298 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33679 every 8 connection(s)
I20260620 14:20:29.062259 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:29.064553 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.051104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52329 (local address 127.0.98.62:45371)
0620 14:20:29.051229 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.051232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.052352 (+ 1120us) server_negotiation.cc:407] Connection header received
0620 14:20:29.053477 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.053481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.053566 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.053632 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:29.055447 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.056396 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.057749 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.057972 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.058090 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.060837 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:29.060841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:29.060844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:29.060893 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:29.062639 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:29.063228 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:29.063231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:29.063232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:29.063309 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:29.063610 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:29.063612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:29.063613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:29.063816 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:29.063907 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.064326 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.064424 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:20:29.065338 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.051681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45371 (local address 127.0.98.3:52329)
0620 14:20:29.052197 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.052240 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.053195 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.054105 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.054117 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.054546 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:29.055272 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.055291 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.056531 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.056537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.057618 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.057627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.058144 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.058149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.058331 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.058954 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:29.058969 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:29.060654 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:29.062788 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:29.062793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:29.062807 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:29.063124 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:29.063394 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:29.063396 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:29.063397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:29.063529 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:29.063904 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:29.063909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:29.064123 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.064273 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.064508 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
I20260620 14:20:29.066522 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45371
I20260620 14:20:29.066665 2605 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:29.067181 2605 heartbeater.cc:511] Master 127.0.98.62:45371 requested a full tablet report, sending...
I20260620 14:20:29.068369 2122 ts_manager.cc:194] Registered new tserver with Master: 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679)
I20260620 14:20:29.069137 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52329
I20260620 14:20:29.071803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 2474
I20260620 14:20:29.071920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:29.072219 392 external_mini_cluster.cc:1428] Setting key a6fb94ca6f9ff804d6046476ff7e5919
I20260620 14:20:29.074404 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:20:29.076233 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.068824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48711 (local address 127.0.98.62:45371)
0620 14:20:29.068955 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.068959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.069093 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:20:29.069185 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.069191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.069257 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.069337 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:29.069854 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.070592 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.071355 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.071657 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.071796 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.073055 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:29.073058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:29.073061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:29.073101 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:29.074649 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:29.075089 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:29.075092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:29.075093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:29.075182 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:29.075467 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:29.075469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:29.075470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:29.075633 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:29.075705 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.076029 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.076126 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:20:29.076419 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.068719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45371 (local address 127.0.98.3:48711)
0620 14:20:29.068992 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.069006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.069108 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.069371 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.069374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.069614 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:29.069751 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.069757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.070705 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.070707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.071257 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.071264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.071814 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.071820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.071931 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.072449 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:29.072463 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:29.072877 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:29.074747 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:29.074750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:29.074752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:29.074992 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:29.075286 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:29.075292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:29.075296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:29.075386 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:29.075722 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:29.075727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:29.075866 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.075979 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.076150 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
Jun 20 14:20:29 dist-test-slave-87nc krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:20:29 dist-test-slave-87nc krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:29.099941 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.091932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.0.98.62:45371)
0620 14:20:29.092060 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.092063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.092081 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:20:29.092117 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.092120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.092155 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.092242 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:29.092916 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.093435 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.094310 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.094531 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.094642 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.097090 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:29.097094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:29.097096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:29.097119 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:29.098289 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:29.098699 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:29.098704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:29.098706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:29.098785 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:29.099060 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:29.099064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:29.099073 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:29.099243 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:29.099383 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.099611 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.099721 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:20:29.102643 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:29.105810 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:29.120018 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.114345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32787 (local address 127.0.0.1:39244)
0620 14:20:29.114982 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.114995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.115091 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.116298 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.116301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.116317 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:29.116424 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.116430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.117489 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.117492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.118526 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.118541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.119497 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.119499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.119589 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.119618 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.119737 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.119871 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260620 14:20:29.120024 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.114526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.0.98.1:32787)
0620 14:20:29.115939 (+ 1413us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.115944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.115966 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:20:29.116036 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.116040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.116181 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.116319 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:29.116539 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.117264 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.118655 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.119387 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.119495 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.119637 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.119792 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.119884 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":97,"threads_started":1}
I20260620 14:20:29.120221 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.114055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.0.1:50510)
0620 14:20:29.114708 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.114720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.114833 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.115630 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.115633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.115673 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:29.115923 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.115930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.117250 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.117253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.118780 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.118790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.119816 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.119820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.119920 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.119942 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.119991 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.120055 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":258,"spinlock_wait_cycles":18176,"thread_start_us":109,"threads_started":1}
I20260620 14:20:29.120923 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.114629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.0.98.3:33679)
0620 14:20:29.115194 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.115206 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.115227 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:20:29.115298 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.115305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.115473 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.115658 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:29.116029 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.117143 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.118902 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.119662 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.120496 (+ 834us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.120667 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.120716 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.120798 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":94,"threads_started":1}
I20260620 14:20:29.121711 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.113906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43971 (local address 127.0.0.1:39022)
0620 14:20:29.114658 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.114704 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.114833 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.115624 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.115630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.115670 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:29.115836 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.115843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.117278 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.117281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.118780 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.118790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.121342 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.121345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.121427 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.121449 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.121525 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.121593 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":90,"spinlock_wait_cycles":11904,"thread_start_us":61,"threads_started":1}
I20260620 14:20:29.121918 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.114049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.0.98.2:43971)
0620 14:20:29.115133 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.115146 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.115172 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:20:29.115248 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.115255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.115461 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.115636 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:29.115947 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.117134 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.119711 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.121226 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.121331 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.121458 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.121547 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.121633 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":134,"threads_started":1}
I20260620 14:20:29.123136 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 482dcb6555db47b6b89d73bd349d43f4 (DEFAULT_TABLE table=test-table [id=b87c3351b020455982589a8ccf00caa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:29.123390 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 482dcb6555db47b6b89d73bd349d43f4 (DEFAULT_TABLE table=test-table [id=b87c3351b020455982589a8ccf00caa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:29.124171 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482dcb6555db47b6b89d73bd349d43f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:29.124390 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482dcb6555db47b6b89d73bd349d43f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:29.124577 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 482dcb6555db47b6b89d73bd349d43f4 (DEFAULT_TABLE table=test-table [id=b87c3351b020455982589a8ccf00caa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:29.125411 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482dcb6555db47b6b89d73bd349d43f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:29.130445 2628 tablet_bootstrap.cc:492] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: Bootstrap starting.
I20260620 14:20:29.132124 2629 tablet_bootstrap.cc:492] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: Bootstrap starting.
I20260620 14:20:29.132761 2628 tablet_bootstrap.cc:654] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:29.132875 2630 tablet_bootstrap.cc:492] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: Bootstrap starting.
I20260620 14:20:29.133697 2628 log.cc:824] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:29.134178 2629 tablet_bootstrap.cc:654] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:29.134599 2630 tablet_bootstrap.cc:654] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:29.135025 2629 log.cc:824] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:29.135421 2630 log.cc:824] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:29.135933 2628 tablet_bootstrap.cc:492] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: No bootstrap required, opened a new log
I20260620 14:20:29.136118 2628 ts_tablet_manager.cc:1397] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:20:29.137007 2630 tablet_bootstrap.cc:492] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: No bootstrap required, opened a new log
I20260620 14:20:29.137022 2629 tablet_bootstrap.cc:492] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: No bootstrap required, opened a new log
I20260620 14:20:29.137123 2630 ts_tablet_manager.cc:1397] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:20:29.137192 2629 ts_tablet_manager.cc:1397] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:20:29.139667 2630 raft_consensus.cc:348] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } }
I20260620 14:20:29.139954 2630 raft_consensus.cc:374] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:29.140007 2630 raft_consensus.cc:729] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a325f95b734d4a128a4ab18d88c140ce, State: Initialized, Role: FOLLOWER
I20260620 14:20:29.140424 2630 consensus_queue.cc:260] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } }
I20260620 14:20:29.140332 2628 raft_consensus.cc:348] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } }
I20260620 14:20:29.140580 2628 raft_consensus.cc:374] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:29.140642 2628 raft_consensus.cc:729] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58b25134e6448b4b6b26eed83bc2bbe, State: Initialized, Role: FOLLOWER
I20260620 14:20:29.141052 2465 heartbeater.cc:503] Master 127.0.98.62:45371 was elected leader, sending a full tablet report...
I20260620 14:20:29.140913 2629 raft_consensus.cc:348] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } }
I20260620 14:20:29.141047 2628 consensus_queue.cc:260] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } }
I20260620 14:20:29.141134 2629 raft_consensus.cc:374] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:29.141180 2629 raft_consensus.cc:729] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db0ca64ef454dceacd54070fb218bd0, State: Initialized, Role: FOLLOWER
I20260620 14:20:29.141583 2629 consensus_queue.cc:260] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } }
I20260620 14:20:29.141949 2628 ts_tablet_manager.cc:1428] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:20:29.142277 2325 heartbeater.cc:503] Master 127.0.98.62:45371 was elected leader, sending a full tablet report...
I20260620 14:20:29.142691 2629 ts_tablet_manager.cc:1428] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:20:29.143016 2605 heartbeater.cc:503] Master 127.0.98.62:45371 was elected leader, sending a full tablet report...
I20260620 14:20:29.143313 2630 ts_tablet_manager.cc:1428] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 14:20:29.313344 2606 tablet.cc:2369] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:29.340492 2466 tablet.cc:2369] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:29.348783 2635 raft_consensus.cc:482] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:29.349011 2635 raft_consensus.cc:504] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } }
I20260620 14:20:29.350317 2635 leader_election.cc:288] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a325f95b734d4a128a4ab18d88c140ce (127.0.98.2:43971), 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679)
I20260620 14:20:29.354404 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.350731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54807 (local address 127.0.98.2:43971)
0620 14:20:29.350849 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.350853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.351072 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:20:29.351223 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.351226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.351268 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.351361 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:29.351693 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.352239 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.353415 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.353814 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.353924 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.353992 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.354187 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.354267 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:20:29.354413 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.350648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39735 (local address 127.0.98.3:33679)
0620 14:20:29.350796 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.350801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.351005 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:20:29.351187 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.351191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.351249 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.351306 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:29.351658 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.352234 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.353443 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.353832 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.353954 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.354019 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.354230 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.354308 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:20:29.354439 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.350645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43971 (local address 127.0.98.1:54807)
0620 14:20:29.350989 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.351000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.351096 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.351384 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.351387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.351423 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:29.351545 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.351550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.352374 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.352376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.353283 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.353291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.353942 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.353946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.354052 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.354079 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.354139 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.354229 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":40,"thread_start_us":97,"threads_started":1}
I20260620 14:20:29.354445 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.350547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.98.1:39735)
0620 14:20:29.350911 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.350924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.351058 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.351345 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.351351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.351367 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:29.351530 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.351536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.352340 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.352345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.353292 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.353301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.353969 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.353973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.354085 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.354103 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.354200 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.354267 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":71,"spinlock_wait_cycles":25344,"thread_start_us":105,"threads_started":1}
I20260620 14:20:29.355173 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482dcb6555db47b6b89d73bd349d43f4" candidate_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a325f95b734d4a128a4ab18d88c140ce" is_pre_election: true
I20260620 14:20:29.355334 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482dcb6555db47b6b89d73bd349d43f4" candidate_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0ca64ef454dceacd54070fb218bd0" is_pre_election: true
I20260620 14:20:29.355535 2419 raft_consensus.cc:2454] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a58b25134e6448b4b6b26eed83bc2bbe in term 0.
I20260620 14:20:29.355646 2559 raft_consensus.cc:2454] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a58b25134e6448b4b6b26eed83bc2bbe in term 0.
I20260620 14:20:29.356129 2211 leader_election.cc:302] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4db0ca64ef454dceacd54070fb218bd0, a325f95b734d4a128a4ab18d88c140ce, a58b25134e6448b4b6b26eed83bc2bbe; no voters:
I20260620 14:20:29.356442 2635 raft_consensus.cc:2781] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:29.356582 2635 raft_consensus.cc:482] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:29.356647 2635 raft_consensus.cc:3037] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:29.357695 2635 raft_consensus.cc:504] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } }
I20260620 14:20:29.358080 2635 leader_election.cc:288] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [CANDIDATE]: Term 1 election: Requested vote from peers a325f95b734d4a128a4ab18d88c140ce (127.0.98.2:43971), 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679)
I20260620 14:20:29.358379 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482dcb6555db47b6b89d73bd349d43f4" candidate_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a325f95b734d4a128a4ab18d88c140ce"
I20260620 14:20:29.358393 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482dcb6555db47b6b89d73bd349d43f4" candidate_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0ca64ef454dceacd54070fb218bd0"
I20260620 14:20:29.358518 2419 raft_consensus.cc:3037] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:29.358530 2559 raft_consensus.cc:3037] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:29.359483 2419 raft_consensus.cc:2454] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a58b25134e6448b4b6b26eed83bc2bbe in term 1.
I20260620 14:20:29.359859 2211 leader_election.cc:302] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a325f95b734d4a128a4ab18d88c140ce, a58b25134e6448b4b6b26eed83bc2bbe; no voters:
I20260620 14:20:29.360049 2635 raft_consensus.cc:2781] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:29.360137 2559 raft_consensus.cc:2454] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a58b25134e6448b4b6b26eed83bc2bbe in term 1.
I20260620 14:20:29.360327 2635 raft_consensus.cc:686] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 LEADER]: Becoming Leader. State: Replica: a58b25134e6448b4b6b26eed83bc2bbe, State: Running, Role: LEADER
W20260620 14:20:29.360627 2326 tablet.cc:2369] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:29.360697 2635 consensus_queue.cc:237] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } }
I20260620 14:20:29.363759 2122 catalog_manager.cc:5495] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe reported cstate change: term changed from 0 to 1, leader changed from <none> to a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } health_report { overall_health: HEALTHY } } }
Jun 20 14:20:29 dist-test-slave-87nc krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:29.433429 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.425637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.0.98.62:45371)
0620 14:20:29.425781 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.425786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.425966 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:20:29.426078 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.426080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.426122 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.426208 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:29.426877 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.427402 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.428316 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.428516 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.428598 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.430735 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:29.430738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:29.430740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:29.430759 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:29.431992 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:29.432405 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:29.432407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:29.432408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:29.432472 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:29.432706 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:29.432708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:29.432709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:29.432857 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:29.432927 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.433186 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.433303 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":60}
I20260620 14:20:29.433431 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.425588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45371 (local address 127.0.0.1:46978)
0620 14:20:29.425872 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.425885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.425991 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.426252 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.426255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.426686 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:29.426776 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.426782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.427564 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.427569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.428215 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.428223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.428608 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.428610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.428676 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.429195 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:29.429206 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:29.430616 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:29.432088 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:29.432096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:29.432102 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:29.432324 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:29.432546 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:29.432548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:29.432549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:29.432633 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:29.432940 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:29.432944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:29.433039 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.433136 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.433251 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":121,"threads_started":1}
I20260620 14:20:29.437193 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46978:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:20:29.437866 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:29.442551 2259 tablet_service.cc:1467] Processing CreateTablet for tablet d67d9b98cf3d4e2aa5b39c51365e3121 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7166eb130e204ce4b64a6e943ac869eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:20:29.442714 2539 tablet_service.cc:1467] Processing CreateTablet for tablet d67d9b98cf3d4e2aa5b39c51365e3121 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7166eb130e204ce4b64a6e943ac869eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:20:29.442600 2399 tablet_service.cc:1467] Processing CreateTablet for tablet d67d9b98cf3d4e2aa5b39c51365e3121 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7166eb130e204ce4b64a6e943ac869eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:20:29.443274 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67d9b98cf3d4e2aa5b39c51365e3121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:29.443284 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67d9b98cf3d4e2aa5b39c51365e3121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:29.443274 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67d9b98cf3d4e2aa5b39c51365e3121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:29.445909 2628 tablet_bootstrap.cc:492] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: Bootstrap starting.
I20260620 14:20:29.446413 2630 tablet_bootstrap.cc:492] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: Bootstrap starting.
I20260620 14:20:29.446641 2629 tablet_bootstrap.cc:492] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: Bootstrap starting.
I20260620 14:20:29.447268 2628 tablet_bootstrap.cc:654] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:29.447443 2630 tablet_bootstrap.cc:654] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:29.447631 2629 tablet_bootstrap.cc:654] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:29.448532 2628 tablet_bootstrap.cc:492] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: No bootstrap required, opened a new log
I20260620 14:20:29.448632 2628 ts_tablet_manager.cc:1397] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:20:29.448936 2629 tablet_bootstrap.cc:492] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: No bootstrap required, opened a new log
I20260620 14:20:29.448994 2629 ts_tablet_manager.cc:1397] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:29.449065 2630 tablet_bootstrap.cc:492] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: No bootstrap required, opened a new log
I20260620 14:20:29.449146 2630 ts_tablet_manager.cc:1397] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 14:20:29.449117 2628 raft_consensus.cc:348] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.449231 2628 raft_consensus.cc:374] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:29.449271 2628 raft_consensus.cc:729] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58b25134e6448b4b6b26eed83bc2bbe, State: Initialized, Role: FOLLOWER
I20260620 14:20:29.449332 2629 raft_consensus.cc:348] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.449402 2628 consensus_queue.cc:260] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.449478 2629 raft_consensus.cc:374] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:29.449525 2629 raft_consensus.cc:729] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db0ca64ef454dceacd54070fb218bd0, State: Initialized, Role: FOLLOWER
I20260620 14:20:29.449590 2630 raft_consensus.cc:348] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.449643 2629 consensus_queue.cc:260] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.449697 2630 raft_consensus.cc:374] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:29.449668 2635 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.449733 2630 raft_consensus.cc:729] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a325f95b734d4a128a4ab18d88c140ce, State: Initialized, Role: FOLLOWER
I20260620 14:20:29.449797 2635 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:29.449860 2630 consensus_queue.cc:260] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.449934 2629 ts_tablet_manager.cc:1428] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:29.449905 2636 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.450022 2636 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:29.450109 2630 ts_tablet_manager.cc:1428] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:20:29.450374 2634 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.450517 2634 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:29.450747 2628 ts_tablet_manager.cc:1428] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:29.702418 2636 raft_consensus.cc:482] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:29.702602 2636 raft_consensus.cc:504] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.703816 2636 leader_election.cc:288] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787), a325f95b734d4a128a4ab18d88c140ce (127.0.98.2:43971)
I20260620 14:20:29.707795 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.704073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43971 (local address 127.0.98.3:44543)
0620 14:20:29.704440 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.704455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.704573 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.704831 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.704834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.704864 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:29.704985 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.704990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.705731 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.705737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.706708 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.706717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.707352 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.707356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.707449 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.707472 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.707547 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.707619 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":128,"thread_start_us":63,"threads_started":1}
I20260620 14:20:29.707814 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.704187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44543 (local address 127.0.98.2:43971)
0620 14:20:29.704315 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.704320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.704553 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:20:29.704666 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.704669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.704708 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.704809 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:29.705120 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.705613 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.706839 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.707231 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.707346 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.707410 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.707583 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.707663 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:20:29.707903 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.704284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45881 (local address 127.0.98.1:32787)
0620 14:20:29.704543 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.704547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.704562 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:29.704619 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.704624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.704676 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.704751 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:29.705177 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.705659 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.706836 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.707243 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.707362 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.707426 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.707660 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.707731 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260620 14:20:29.707829 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.704068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32787 (local address 127.0.98.3:45881)
0620 14:20:29.704376 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.704389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.704487 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.704789 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.704795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.704813 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:29.705001 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.705006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.705797 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.705799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.706704 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.706713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.707384 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.707387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.707502 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.707538 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.707624 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.707683 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":38,"thread_start_us":68,"threads_started":1}
I20260620 14:20:29.708413 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67d9b98cf3d4e2aa5b39c51365e3121" candidate_uuid: "4db0ca64ef454dceacd54070fb218bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a325f95b734d4a128a4ab18d88c140ce" is_pre_election: true
I20260620 14:20:29.708572 2419 raft_consensus.cc:2454] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db0ca64ef454dceacd54070fb218bd0 in term 0.
I20260620 14:20:29.708666 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67d9b98cf3d4e2aa5b39c51365e3121" candidate_uuid: "4db0ca64ef454dceacd54070fb218bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" is_pre_election: true
I20260620 14:20:29.708909 2279 raft_consensus.cc:2454] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db0ca64ef454dceacd54070fb218bd0 in term 0.
I20260620 14:20:29.708953 2491 leader_election.cc:302] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db0ca64ef454dceacd54070fb218bd0, a325f95b734d4a128a4ab18d88c140ce; no voters:
I20260620 14:20:29.709273 2636 raft_consensus.cc:2781] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:29.709342 2636 raft_consensus.cc:482] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:29.709390 2636 raft_consensus.cc:3037] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:29.710321 2636 raft_consensus.cc:504] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.710669 2636 leader_election.cc:288] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [CANDIDATE]: Term 1 election: Requested vote from peers a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787), a325f95b734d4a128a4ab18d88c140ce (127.0.98.2:43971)
I20260620 14:20:29.711030 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67d9b98cf3d4e2aa5b39c51365e3121" candidate_uuid: "4db0ca64ef454dceacd54070fb218bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a325f95b734d4a128a4ab18d88c140ce"
I20260620 14:20:29.711031 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67d9b98cf3d4e2aa5b39c51365e3121" candidate_uuid: "4db0ca64ef454dceacd54070fb218bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58b25134e6448b4b6b26eed83bc2bbe"
I20260620 14:20:29.711159 2419 raft_consensus.cc:3037] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:29.711174 2279 raft_consensus.cc:3037] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:29.712133 2419 raft_consensus.cc:2454] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db0ca64ef454dceacd54070fb218bd0 in term 1.
I20260620 14:20:29.712250 2279 raft_consensus.cc:2454] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db0ca64ef454dceacd54070fb218bd0 in term 1.
I20260620 14:20:29.712414 2491 leader_election.cc:302] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db0ca64ef454dceacd54070fb218bd0, a325f95b734d4a128a4ab18d88c140ce; no voters:
I20260620 14:20:29.712572 2636 raft_consensus.cc:2781] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:29.712736 2636 raft_consensus.cc:686] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4db0ca64ef454dceacd54070fb218bd0, State: Running, Role: LEADER
I20260620 14:20:29.713136 2636 consensus_queue.cc:237] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.715515 2653 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: TxnStatusTablet state changed. Reason: New leader 4db0ca64ef454dceacd54070fb218bd0. Latest consensus state: current_term: 1 leader_uuid: "4db0ca64ef454dceacd54070fb218bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.715633 2653 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:20:29.715909 2655 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:20:29.716107 2655 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:20:29.716470 2121 catalog_manager.cc:5495] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4db0ca64ef454dceacd54070fb218bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: HEALTHY } } }
I20260620 14:20:29.747802 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.743448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.0.98.3:33679)
0620 14:20:29.743577 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.743582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.743596 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:20:29.743820 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.743824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.743892 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.744000 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:29.744263 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.745064 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.746021 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.746391 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.746517 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.746821 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.747367 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.747487 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.747576 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.747631 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:20:29.747822 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.743312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.0.1:50514)
0620 14:20:29.743467 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.743485 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.743611 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.744025 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.744030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.744058 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:29.744164 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.744169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.745196 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.745199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.745912 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.745922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.746536 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.746539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.746656 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.746702 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.747486 (+ 784us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.747507 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.747562 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.747634 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:20:29.753448 2419 raft_consensus.cc:1261] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Refusing update from remote peer 4db0ca64ef454dceacd54070fb218bd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:29.753566 2279 raft_consensus.cc:1261] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Refusing update from remote peer 4db0ca64ef454dceacd54070fb218bd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:29.754117 2653 consensus_queue.cc:1035] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:29.754281 2636 consensus_queue.cc:1035] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:29.756987 2639 consensus_queue.cc:1035] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:29.757625 2635 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: TxnStatusTablet state changed. Reason: New leader 4db0ca64ef454dceacd54070fb218bd0. Latest consensus state: current_term: 1 leader_uuid: "4db0ca64ef454dceacd54070fb218bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.757730 2635 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:29.758730 2634 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: TxnStatusTablet state changed. Reason: New leader 4db0ca64ef454dceacd54070fb218bd0. Latest consensus state: current_term: 1 leader_uuid: "4db0ca64ef454dceacd54070fb218bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.758839 2634 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:29.763094 2664 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db0ca64ef454dceacd54070fb218bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.763216 2664 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:20:29.763338 2664 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db0ca64ef454dceacd54070fb218bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.763403 2664 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P 4db0ca64ef454dceacd54070fb218bd0: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:20:29.763849 2660 mvcc.cc:205] Tried to move back new op lower bound from 7298929581063577600 to 7298929580907732992. Current Snapshot: MvccSnapshot[applied={T|T < 7298929581063577600}]
I20260620 14:20:29.764014 2634 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db0ca64ef454dceacd54070fb218bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.764097 2634 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P a325f95b734d4a128a4ab18d88c140ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:29.765255 2639 tablet_replica.cc:439] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db0ca64ef454dceacd54070fb218bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260620 14:20:29.765375 2639 tablet_replica.cc:442] T d67d9b98cf3d4e2aa5b39c51365e3121 P a58b25134e6448b4b6b26eed83bc2bbe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:29.766083 2639 consensus_queue.cc:1035] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:29.766243 2668 mvcc.cc:205] Tried to move back new op lower bound from 7298929581063577600 to 7298929580907732992. Current Snapshot: MvccSnapshot[applied={T|T < 7298929581063577600}]
I20260620 14:20:29.775995 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.771954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.0.98.1:32787)
0620 14:20:29.772071 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.772076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.772248 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:20:29.772458 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.772462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.772542 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.772612 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:29.772982 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.773470 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.774690 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.774888 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.774959 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.775237 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.775624 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.775707 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.775841 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.775898 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:20:29.786489 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.783011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.98.1:54037)
0620 14:20:29.783299 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.783316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.783451 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.783718 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.783720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.783728 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:29.783880 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.783886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.784628 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.784631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.785263 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.785271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.785744 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.785747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.785842 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.785880 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.786202 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.786210 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.786275 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.786352 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
I20260620 14:20:29.786464 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.783168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54037 (local address 127.0.98.3:33679)
0620 14:20:29.783291 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.783295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.783397 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:20:29.783553 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.783556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.783627 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.783710 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:29.783987 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.784495 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.785383 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.785619 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.785731 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.786034 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.786118 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.786199 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.786283 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.786327 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:20:29.815439 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.808564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32787 (local address 127.0.98.3:34531)
0620 14:20:29.812005 (+ 3441us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.812020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.812124 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.812333 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.812336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.812344 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:29.812460 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.812467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.813144 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.813147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.814142 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.814153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.814683 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.814686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.814779 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.814828 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.815180 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.815186 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.815255 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.815315 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3319,"thread_start_us":97,"threads_started":1}
I20260620 14:20:29.815449 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.808652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34531 (local address 127.0.98.1:32787)
0620 14:20:29.808766 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.808772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.812098 (+ 3326us) server_negotiation.cc:407] Connection header received
0620 14:20:29.812202 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.812204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.812243 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.812304 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:29.812559 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.813043 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.814257 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.814565 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.814683 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.815015 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.815099 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.815175 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.815264 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.815312 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:20:29.844717 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:29.844909 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:29.847688 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f: Sending DeleteTablet for 3 replicas of tablet 482dcb6555db47b6b89d73bd349d43f4
I20260620 14:20:29.848587 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 482dcb6555db47b6b89d73bd349d43f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39022
I20260620 14:20:29.848584 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 482dcb6555db47b6b89d73bd349d43f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39244
I20260620 14:20:29.849380 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 482dcb6555db47b6b89d73bd349d43f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260620 14:20:29.849828 2682 tablet_replica.cc:330] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: stopping tablet replica
I20260620 14:20:29.850118 2682 raft_consensus.cc:2229] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:29.850279 2682 raft_consensus.cc:2258] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:29.850240 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:29.850660 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:29.851519 2682 ts_tablet_manager.cc:1905] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:29.854503 2682 ts_tablet_manager.cc:1918] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:29.854611 2682 log.cc:1196] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/482dcb6555db47b6b89d73bd349d43f4
I20260620 14:20:29.855067 2682 ts_tablet_manager.cc:1939] T 482dcb6555db47b6b89d73bd349d43f4 P 4db0ca64ef454dceacd54070fb218bd0: Deleting consensus metadata
I20260620 14:20:29.855919 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 4ecf4a294942411c9c16a3848a81eea4 (DEFAULT_TABLE table=test-table [id=f978d382881449199b433548eb65ad25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:29.855937 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 4ecf4a294942411c9c16a3848a81eea4 (DEFAULT_TABLE table=test-table [id=f978d382881449199b433548eb65ad25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:29.856189 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ecf4a294942411c9c16a3848a81eea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:29.856200 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ecf4a294942411c9c16a3848a81eea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:29.857195 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 4ecf4a294942411c9c16a3848a81eea4 (DEFAULT_TABLE table=test-table [id=f978d382881449199b433548eb65ad25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:29.857493 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ecf4a294942411c9c16a3848a81eea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:29.858444 2630 tablet_bootstrap.cc:492] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: Bootstrap starting.
I20260620 14:20:29.859401 2630 tablet_bootstrap.cc:654] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:29.859652 2629 tablet_bootstrap.cc:492] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: Bootstrap starting.
I20260620 14:20:29.860004 2109 catalog_manager.cc:4841] TS 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679): tablet 482dcb6555db47b6b89d73bd349d43f4 (table test-table [id=b87c3351b020455982589a8ccf00caa5]) successfully deleted
I20260620 14:20:29.860540 2629 tablet_bootstrap.cc:654] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:29.862918 2628 tablet_bootstrap.cc:492] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: Bootstrap starting.
I20260620 14:20:29.863859 2628 tablet_bootstrap.cc:654] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:29.865023 2628 tablet_bootstrap.cc:492] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: No bootstrap required, opened a new log
I20260620 14:20:29.865114 2628 ts_tablet_manager.cc:1397] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:20:29.865566 2628 raft_consensus.cc:348] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.865693 2628 raft_consensus.cc:374] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:29.865736 2628 raft_consensus.cc:729] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58b25134e6448b4b6b26eed83bc2bbe, State: Initialized, Role: FOLLOWER
I20260620 14:20:29.865859 2628 consensus_queue.cc:260] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.866144 2628 ts_tablet_manager.cc:1428] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:20:29.867285 2680 tablet_replica.cc:330] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: stopping tablet replica
I20260620 14:20:29.867930 2680 raft_consensus.cc:2229] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:29.868511 2681 tablet_replica.cc:330] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: stopping tablet replica
I20260620 14:20:29.868834 2681 raft_consensus.cc:2229] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:29.869051 2681 raft_consensus.cc:2258] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:29.869169 2680 raft_consensus.cc:2258] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:29.870980 2681 ts_tablet_manager.cc:1905] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:29.871989 2680 ts_tablet_manager.cc:1905] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:29.872473 2630 tablet_bootstrap.cc:492] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: No bootstrap required, opened a new log
I20260620 14:20:29.872570 2630 ts_tablet_manager.cc:1397] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20260620 14:20:29.872980 2630 raft_consensus.cc:348] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.873101 2630 raft_consensus.cc:374] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:29.873137 2630 raft_consensus.cc:729] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a325f95b734d4a128a4ab18d88c140ce, State: Initialized, Role: FOLLOWER
I20260620 14:20:29.873260 2630 consensus_queue.cc:260] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.873533 2630 ts_tablet_manager.cc:1428] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:29.875322 2680 ts_tablet_manager.cc:1918] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:29.875465 2680 log.cc:1196] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/482dcb6555db47b6b89d73bd349d43f4
I20260620 14:20:29.875913 2680 ts_tablet_manager.cc:1939] T 482dcb6555db47b6b89d73bd349d43f4 P a58b25134e6448b4b6b26eed83bc2bbe: Deleting consensus metadata
I20260620 14:20:29.877151 2106 catalog_manager.cc:4841] TS a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787): tablet 482dcb6555db47b6b89d73bd349d43f4 (table test-table [id=b87c3351b020455982589a8ccf00caa5]) successfully deleted
I20260620 14:20:29.881079 2681 ts_tablet_manager.cc:1918] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:29.881242 2681 log.cc:1196] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/482dcb6555db47b6b89d73bd349d43f4
I20260620 14:20:29.881628 2681 ts_tablet_manager.cc:1939] T 482dcb6555db47b6b89d73bd349d43f4 P a325f95b734d4a128a4ab18d88c140ce: Deleting consensus metadata
I20260620 14:20:29.881748 2629 tablet_bootstrap.cc:492] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: No bootstrap required, opened a new log
I20260620 14:20:29.881848 2629 ts_tablet_manager.cc:1397] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.000s
I20260620 14:20:29.882265 2629 raft_consensus.cc:348] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.882380 2629 raft_consensus.cc:374] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:29.882423 2629 raft_consensus.cc:729] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db0ca64ef454dceacd54070fb218bd0, State: Initialized, Role: FOLLOWER
I20260620 14:20:29.882565 2629 consensus_queue.cc:260] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.882833 2629 ts_tablet_manager.cc:1428] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:29.883245 2107 catalog_manager.cc:4841] TS a325f95b734d4a128a4ab18d88c140ce (127.0.98.2:43971): tablet 482dcb6555db47b6b89d73bd349d43f4 (table test-table [id=b87c3351b020455982589a8ccf00caa5]) successfully deleted
I20260620 14:20:29.902346 2634 raft_consensus.cc:482] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:29.902480 2634 raft_consensus.cc:504] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.903548 2634 leader_election.cc:288] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787), 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679)
I20260620 14:20:29.908469 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.903738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32787 (local address 127.0.98.2:51363)
0620 14:20:29.904104 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.904118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.904219 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.904549 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.904552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.904581 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:29.904710 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.904716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.905532 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.905538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.907051 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.907058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.908022 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.908026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.908125 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.908156 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.908234 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.908302 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":40,"thread_start_us":79,"threads_started":1}
I20260620 14:20:29.908475 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.904093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51363 (local address 127.0.98.1:32787)
0620 14:20:29.904205 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.904209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.904224 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:29.904314 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.904317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.904363 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.904436 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:29.904844 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.905423 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.907184 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.907886 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.907996 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.908062 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.908262 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.908342 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:20:29.909034 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecf4a294942411c9c16a3848a81eea4" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" is_pre_election: true
I20260620 14:20:29.909200 2279 raft_consensus.cc:2454] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 0.
I20260620 14:20:29.909260 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.903897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46733 (local address 127.0.98.3:33679)
0620 14:20:29.904018 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.904023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.904246 (+ 223us) server_negotiation.cc:407] Connection header received
0620 14:20:29.904394 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.904397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.904443 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.904527 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:29.904881 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.905626 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.907865 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.908462 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.908563 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.908627 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.909076 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.909149 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:20:29.909554 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.903758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.98.2:46733)
0620 14:20:29.904180 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.904191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.904302 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.904576 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.904579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.904597 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:29.904723 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.904728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.905753 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.905756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.907047 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.907058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.908832 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.908836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.908940 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.908966 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.909042 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.909112 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":70,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260620 14:20:29.910084 2350 leader_election.cc:302] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a325f95b734d4a128a4ab18d88c140ce, a58b25134e6448b4b6b26eed83bc2bbe; no voters:
I20260620 14:20:29.910075 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecf4a294942411c9c16a3848a81eea4" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0ca64ef454dceacd54070fb218bd0" is_pre_election: true
I20260620 14:20:29.910233 2559 raft_consensus.cc:2454] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 0.
I20260620 14:20:29.910346 2634 raft_consensus.cc:2781] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:29.910431 2634 raft_consensus.cc:482] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:29.910476 2634 raft_consensus.cc:3037] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:29.911803 2634 raft_consensus.cc:504] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.912181 2634 leader_election.cc:288] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 election: Requested vote from peers a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787), 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679)
I20260620 14:20:29.912479 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecf4a294942411c9c16a3848a81eea4" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0ca64ef454dceacd54070fb218bd0"
I20260620 14:20:29.912581 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecf4a294942411c9c16a3848a81eea4" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58b25134e6448b4b6b26eed83bc2bbe"
I20260620 14:20:29.912700 2279 raft_consensus.cc:3037] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:29.913952 2279 raft_consensus.cc:2454] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 1.
I20260620 14:20:29.914227 2350 leader_election.cc:302] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a325f95b734d4a128a4ab18d88c140ce, a58b25134e6448b4b6b26eed83bc2bbe; no voters:
I20260620 14:20:29.914430 2634 raft_consensus.cc:2781] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:29.914472 2559 raft_consensus.cc:3037] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:29.914621 2634 raft_consensus.cc:686] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 1 LEADER]: Becoming Leader. State: Replica: a325f95b734d4a128a4ab18d88c140ce, State: Running, Role: LEADER
I20260620 14:20:29.914867 2634 consensus_queue.cc:237] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:29.915602 2559 raft_consensus.cc:2454] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 1.
I20260620 14:20:29.917070 2119 catalog_manager.cc:5495] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce reported cstate change: term changed from 0 to 1, leader changed from <none> to a325f95b734d4a128a4ab18d88c140ce (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a325f95b734d4a128a4ab18d88c140ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: UNKNOWN } } }
I20260620 14:20:29.936800 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.933012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.0.98.2:43971)
0620 14:20:29.933127 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.933131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.933146 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:29.933280 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.933283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.933344 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.933456 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:29.933658 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.934430 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.935215 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.935554 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.935657 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.935873 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.936429 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.936542 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.936607 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.936661 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:20:29.947693 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.944050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39445 (local address 127.0.98.3:33679)
0620 14:20:29.944203 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.944210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.944227 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:20:29.944277 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.944279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.944318 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.944381 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:29.944730 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.945340 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.946612 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.946821 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.946899 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.947257 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.947324 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.947382 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.947539 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.947584 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:20:29.947786 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.943735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.98.2:39445)
0620 14:20:29.944064 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.944080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.944196 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.944416 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.944418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.944427 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:29.944557 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.944573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.945441 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.945445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.946496 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.946507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.946934 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.946938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.947034 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.947083 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.947435 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.947442 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.947516 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.947600 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":128,"threads_started":1}
I20260620 14:20:29.953837 2279 raft_consensus.cc:1261] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Refusing update from remote peer a325f95b734d4a128a4ab18d88c140ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:29.953824 2559 raft_consensus.cc:1261] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Refusing update from remote peer a325f95b734d4a128a4ab18d88c140ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:29.954317 2688 consensus_queue.cc:1035] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:29.954684 2688 consensus_queue.cc:1035] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:29.958076 2659 mvcc.cc:205] Tried to move back new op lower bound from 7298929581886783488 to 7298929581733949440. Current Snapshot: MvccSnapshot[applied={T|T < 7298929581886783488}]
I20260620 14:20:29.975311 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.971744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45955 (local address 127.0.98.2:43971)
0620 14:20:29.971887 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:29.971892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:29.971907 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:29.972002 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:29.972005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:29.972052 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:29.972129 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:29.972448 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.972941 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.974176 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.974496 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.974622 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.974861 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.974949 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.975018 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:29.975152 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:20:29.975205 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:20:29.975355 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:29.971546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43971 (local address 127.0.98.3:45955)
0620 14:20:29.971828 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:29.971846 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:29.971918 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:29.972241 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:29.972244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:29.972252 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:29.972354 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.972360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.973046 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.973049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.974032 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:29.974043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:29.974617 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:29.974621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:29.974712 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:29.974755 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:29.975041 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:29.975049 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:20:29.975126 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:20:29.975220 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
W20260620 14:20:29.976150 2660 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:20:31.538645 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:31.538793 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:31.540838 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f: Sending DeleteTablet for 3 replicas of tablet 4ecf4a294942411c9c16a3848a81eea4
I20260620 14:20:31.541358 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ecf4a294942411c9c16a3848a81eea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39244
I20260620 14:20:31.541507 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ecf4a294942411c9c16a3848a81eea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260620 14:20:31.541548 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ecf4a294942411c9c16a3848a81eea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39022
I20260620 14:20:31.541699 2720 tablet_replica.cc:330] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: stopping tablet replica
I20260620 14:20:31.541829 2720 raft_consensus.cc:2229] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:31.542021 2720 raft_consensus.cc:2258] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:31.542719 2720 ts_tablet_manager.cc:1905] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:31.544751 2722 tablet_replica.cc:330] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: stopping tablet replica
I20260620 14:20:31.544950 2722 raft_consensus.cc:2229] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:31.545276 2722 raft_consensus.cc:2258] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:31.545534 2721 tablet_replica.cc:330] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: stopping tablet replica
I20260620 14:20:31.545701 2721 raft_consensus.cc:2229] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:31.545730 2720 ts_tablet_manager.cc:1918] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:20:31.545850 2721 raft_consensus.cc:2258] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:31.545876 2720 log.cc:1196] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/4ecf4a294942411c9c16a3848a81eea4
I20260620 14:20:31.545951 2722 ts_tablet_manager.cc:1905] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:31.546299 2720 ts_tablet_manager.cc:1939] T 4ecf4a294942411c9c16a3848a81eea4 P a58b25134e6448b4b6b26eed83bc2bbe: Deleting consensus metadata
I20260620 14:20:31.546530 2721 ts_tablet_manager.cc:1905] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:31.547173 2106 catalog_manager.cc:4841] TS a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787): tablet 4ecf4a294942411c9c16a3848a81eea4 (table test-table [id=f978d382881449199b433548eb65ad25]) successfully deleted
I20260620 14:20:31.548775 2722 ts_tablet_manager.cc:1918] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:20:31.548914 2722 log.cc:1196] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/4ecf4a294942411c9c16a3848a81eea4
I20260620 14:20:31.549162 2721 ts_tablet_manager.cc:1918] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:20:31.549230 2722 ts_tablet_manager.cc:1939] T 4ecf4a294942411c9c16a3848a81eea4 P a325f95b734d4a128a4ab18d88c140ce: Deleting consensus metadata
I20260620 14:20:31.549242 2721 log.cc:1196] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/4ecf4a294942411c9c16a3848a81eea4
I20260620 14:20:31.549500 2721 ts_tablet_manager.cc:1939] T 4ecf4a294942411c9c16a3848a81eea4 P 4db0ca64ef454dceacd54070fb218bd0: Deleting consensus metadata
I20260620 14:20:31.549984 2107 catalog_manager.cc:4841] TS a325f95b734d4a128a4ab18d88c140ce (127.0.98.2:43971): tablet 4ecf4a294942411c9c16a3848a81eea4 (table test-table [id=f978d382881449199b433548eb65ad25]) successfully deleted
I20260620 14:20:31.550100 2109 catalog_manager.cc:4841] TS 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679): tablet 4ecf4a294942411c9c16a3848a81eea4 (table test-table [id=f978d382881449199b433548eb65ad25]) successfully deleted
Jun 20 14:20:31 dist-test-slave-87nc krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:20:31 dist-test-slave-87nc krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:31.571663 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:31.559978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.0.98.62:45371)
0620 14:20:31.560237 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:31.560244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:31.560259 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:31.560379 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:31.560381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:31.560423 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:31.560499 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:31.561195 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:31.561754 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:31.562601 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:31.562834 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:31.562914 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:31.568934 (+ 6020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:31.568937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:31.568940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:31.568967 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:31.570281 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:31.570649 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:31.570652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:31.570653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:31.570733 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:31.570946 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:31.570948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:31.570949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:31.571128 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:31.571224 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:31.571428 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:20:31.571524 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
I20260620 14:20:31.574136 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:31.574572 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:31.579025 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e (DEFAULT_TABLE table=test-table [id=82f835c5171f46909cb123e924695da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:31.579116 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e (DEFAULT_TABLE table=test-table [id=82f835c5171f46909cb123e924695da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:31.579162 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e (DEFAULT_TABLE table=test-table [id=82f835c5171f46909cb123e924695da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:31.579319 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:31.579392 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:31.579401 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:31.581583 2731 tablet_bootstrap.cc:492] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: Bootstrap starting.
I20260620 14:20:31.582238 2732 tablet_bootstrap.cc:492] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: Bootstrap starting.
I20260620 14:20:31.582505 2731 tablet_bootstrap.cc:654] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:31.583007 2732 tablet_bootstrap.cc:654] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:31.583571 2733 tablet_bootstrap.cc:492] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: Bootstrap starting.
I20260620 14:20:31.583642 2731 tablet_bootstrap.cc:492] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: No bootstrap required, opened a new log
I20260620 14:20:31.583817 2731 ts_tablet_manager.cc:1397] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:31.584159 2732 tablet_bootstrap.cc:492] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: No bootstrap required, opened a new log
I20260620 14:20:31.584247 2732 ts_tablet_manager.cc:1397] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:31.584259 2731 raft_consensus.cc:348] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.584378 2731 raft_consensus.cc:374] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:31.584412 2731 raft_consensus.cc:729] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db0ca64ef454dceacd54070fb218bd0, State: Initialized, Role: FOLLOWER
I20260620 14:20:31.584470 2733 tablet_bootstrap.cc:654] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:31.584578 2731 consensus_queue.cc:260] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.584702 2732 raft_consensus.cc:348] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.584800 2732 raft_consensus.cc:374] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:31.584873 2732 raft_consensus.cc:729] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a325f95b734d4a128a4ab18d88c140ce, State: Initialized, Role: FOLLOWER
I20260620 14:20:31.585065 2732 consensus_queue.cc:260] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.585423 2731 ts_tablet_manager.cc:1428] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:31.585534 2732 ts_tablet_manager.cc:1428] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:31.586303 2733 tablet_bootstrap.cc:492] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: No bootstrap required, opened a new log
I20260620 14:20:31.586405 2733 ts_tablet_manager.cc:1397] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:20:31.586812 2733 raft_consensus.cc:348] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.586930 2733 raft_consensus.cc:374] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:31.587007 2733 raft_consensus.cc:729] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58b25134e6448b4b6b26eed83bc2bbe, State: Initialized, Role: FOLLOWER
I20260620 14:20:31.587143 2733 consensus_queue.cc:260] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.587657 2733 ts_tablet_manager.cc:1428] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:31.801672 2711 raft_consensus.cc:482] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:31.801813 2711 raft_consensus.cc:504] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.802270 2711 leader_election.cc:288] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787), 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679)
I20260620 14:20:31.802735 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f44d5cd8ade46a99d4eb5c94d7eef5e" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0ca64ef454dceacd54070fb218bd0" is_pre_election: true
I20260620 14:20:31.802739 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f44d5cd8ade46a99d4eb5c94d7eef5e" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" is_pre_election: true
I20260620 14:20:31.802870 2279 raft_consensus.cc:2454] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 0.
I20260620 14:20:31.802897 2559 raft_consensus.cc:2454] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 0.
I20260620 14:20:31.803254 2350 leader_election.cc:302] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a325f95b734d4a128a4ab18d88c140ce, a58b25134e6448b4b6b26eed83bc2bbe; no voters:
I20260620 14:20:31.803507 2711 raft_consensus.cc:2781] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:31.803598 2711 raft_consensus.cc:482] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:31.803655 2711 raft_consensus.cc:3037] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:31.804926 2711 raft_consensus.cc:504] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.805266 2711 leader_election.cc:288] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 election: Requested vote from peers a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787), 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679)
I20260620 14:20:31.805668 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f44d5cd8ade46a99d4eb5c94d7eef5e" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0ca64ef454dceacd54070fb218bd0"
I20260620 14:20:31.805668 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f44d5cd8ade46a99d4eb5c94d7eef5e" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58b25134e6448b4b6b26eed83bc2bbe"
I20260620 14:20:31.805801 2279 raft_consensus.cc:3037] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:31.805821 2559 raft_consensus.cc:3037] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:31.806939 2279 raft_consensus.cc:2454] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 1.
I20260620 14:20:31.806939 2559 raft_consensus.cc:2454] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 1.
I20260620 14:20:31.807232 2350 leader_election.cc:302] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a325f95b734d4a128a4ab18d88c140ce, a58b25134e6448b4b6b26eed83bc2bbe; no voters:
I20260620 14:20:31.807389 2711 raft_consensus.cc:2781] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:31.807482 2711 raft_consensus.cc:686] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 1 LEADER]: Becoming Leader. State: Replica: a325f95b734d4a128a4ab18d88c140ce, State: Running, Role: LEADER
I20260620 14:20:31.807621 2711 consensus_queue.cc:237] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.809279 2120 catalog_manager.cc:5495] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce reported cstate change: term changed from 0 to 1, leader changed from <none> to a325f95b734d4a128a4ab18d88c140ce (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a325f95b734d4a128a4ab18d88c140ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: UNKNOWN } } }
I20260620 14:20:31.825706 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:31.822889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.0.98.2:43971)
0620 14:20:31.823132 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:31.823138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:31.823149 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:20:31.823183 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:31.823185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:31.823222 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:31.823291 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:31.823485 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:31.824010 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:31.824800 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:31.825008 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:31.825078 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:31.825282 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:31.825345 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:31.825396 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:31.825470 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:20:31.825516 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
I20260620 14:20:31.831378 2279 raft_consensus.cc:1261] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Refusing update from remote peer a325f95b734d4a128a4ab18d88c140ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:31.831436 2559 raft_consensus.cc:1261] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Refusing update from remote peer a325f95b734d4a128a4ab18d88c140ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:31.831816 2714 consensus_queue.cc:1035] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:31.831966 2711 consensus_queue.cc:1035] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:31.863515 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:31.863677 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:31.865664 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f: Sending DeleteTablet for 3 replicas of tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e
I20260620 14:20:31.866168 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39244
I20260620 14:20:31.866199 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39022
I20260620 14:20:31.866386 2722 tablet_replica.cc:330] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: stopping tablet replica
I20260620 14:20:31.866408 2720 tablet_replica.cc:330] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: stopping tablet replica
I20260620 14:20:31.866393 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260620 14:20:31.866531 2722 raft_consensus.cc:2229] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:31.866561 2720 raft_consensus.cc:2229] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:31.866582 2721 tablet_replica.cc:330] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: stopping tablet replica
I20260620 14:20:31.866676 2720 raft_consensus.cc:2258] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:31.866699 2721 raft_consensus.cc:2229] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:31.866710 2722 raft_consensus.cc:2258] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:31.866804 2721 raft_consensus.cc:2258] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:31.867381 2722 ts_tablet_manager.cc:1905] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:31.867594 2720 ts_tablet_manager.cc:1905] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:31.867846 2721 ts_tablet_manager.cc:1905] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:31.869612 2720 ts_tablet_manager.cc:1918] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:31.869665 2720 log.cc:1196] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/3f44d5cd8ade46a99d4eb5c94d7eef5e
I20260620 14:20:31.869844 2720 ts_tablet_manager.cc:1939] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a58b25134e6448b4b6b26eed83bc2bbe: Deleting consensus metadata
I20260620 14:20:31.870007 2722 ts_tablet_manager.cc:1918] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:31.870080 2722 log.cc:1196] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/3f44d5cd8ade46a99d4eb5c94d7eef5e
I20260620 14:20:31.870245 2721 ts_tablet_manager.cc:1918] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:31.870313 2721 log.cc:1196] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/3f44d5cd8ade46a99d4eb5c94d7eef5e
I20260620 14:20:31.870343 2722 ts_tablet_manager.cc:1939] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P a325f95b734d4a128a4ab18d88c140ce: Deleting consensus metadata
I20260620 14:20:31.870545 2721 ts_tablet_manager.cc:1939] T 3f44d5cd8ade46a99d4eb5c94d7eef5e P 4db0ca64ef454dceacd54070fb218bd0: Deleting consensus metadata
I20260620 14:20:31.871008 2107 catalog_manager.cc:4841] TS a325f95b734d4a128a4ab18d88c140ce (127.0.98.2:43971): tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e (table test-table [id=82f835c5171f46909cb123e924695da4]) successfully deleted
I20260620 14:20:31.871124 2109 catalog_manager.cc:4841] TS 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679): tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e (table test-table [id=82f835c5171f46909cb123e924695da4]) successfully deleted
I20260620 14:20:31.871166 2106 catalog_manager.cc:4841] TS a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787): tablet 3f44d5cd8ade46a99d4eb5c94d7eef5e (table test-table [id=82f835c5171f46909cb123e924695da4]) successfully deleted
I20260620 14:20:31.871568 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:31.871990 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:31.876262 2259 tablet_service.cc:1467] Processing CreateTablet for tablet bf4ea68561e14bf0bcfba08cedfbdfc7 (DEFAULT_TABLE table=test-table [id=b6de389ce94a4d1d840a348791694996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:31.876365 2399 tablet_service.cc:1467] Processing CreateTablet for tablet bf4ea68561e14bf0bcfba08cedfbdfc7 (DEFAULT_TABLE table=test-table [id=b6de389ce94a4d1d840a348791694996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:31.876442 2539 tablet_service.cc:1467] Processing CreateTablet for tablet bf4ea68561e14bf0bcfba08cedfbdfc7 (DEFAULT_TABLE table=test-table [id=b6de389ce94a4d1d840a348791694996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:31.876559 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4ea68561e14bf0bcfba08cedfbdfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:31.876614 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4ea68561e14bf0bcfba08cedfbdfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:31.876644 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4ea68561e14bf0bcfba08cedfbdfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:31.878734 2732 tablet_bootstrap.cc:492] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce: Bootstrap starting.
I20260620 14:20:31.878815 2733 tablet_bootstrap.cc:492] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe: Bootstrap starting.
I20260620 14:20:31.879526 2732 tablet_bootstrap.cc:654] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:31.879647 2733 tablet_bootstrap.cc:654] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:31.880093 2731 tablet_bootstrap.cc:492] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: Bootstrap starting.
I20260620 14:20:31.880648 2732 tablet_bootstrap.cc:492] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce: No bootstrap required, opened a new log
I20260620 14:20:31.880735 2732 ts_tablet_manager.cc:1397] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:31.880934 2731 tablet_bootstrap.cc:654] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:31.881202 2732 raft_consensus.cc:348] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.881318 2732 raft_consensus.cc:374] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:31.881368 2732 raft_consensus.cc:729] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a325f95b734d4a128a4ab18d88c140ce, State: Initialized, Role: FOLLOWER
I20260620 14:20:31.881501 2732 consensus_queue.cc:260] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.881812 2732 ts_tablet_manager.cc:1428] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:20:31.882066 2731 tablet_bootstrap.cc:492] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: No bootstrap required, opened a new log
I20260620 14:20:31.882144 2731 ts_tablet_manager.cc:1397] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:31.882526 2731 raft_consensus.cc:348] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.882634 2731 raft_consensus.cc:374] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:31.882659 2733 tablet_bootstrap.cc:492] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe: No bootstrap required, opened a new log
I20260620 14:20:31.882664 2731 raft_consensus.cc:729] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db0ca64ef454dceacd54070fb218bd0, State: Initialized, Role: FOLLOWER
I20260620 14:20:31.882737 2733 ts_tablet_manager.cc:1397] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 14:20:31.882797 2731 consensus_queue.cc:260] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.883054 2731 ts_tablet_manager.cc:1428] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:20:31.883142 2733 raft_consensus.cc:348] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.883258 2733 raft_consensus.cc:374] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:31.883314 2733 raft_consensus.cc:729] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58b25134e6448b4b6b26eed83bc2bbe, State: Initialized, Role: FOLLOWER
I20260620 14:20:31.883441 2733 consensus_queue.cc:260] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.883750 2733 ts_tablet_manager.cc:1428] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:31.886873 2711 raft_consensus.cc:482] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:31.886945 2711 raft_consensus.cc:504] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.887241 2711 leader_election.cc:288] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787), 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679)
I20260620 14:20:31.887522 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4ea68561e14bf0bcfba08cedfbdfc7" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" is_pre_election: true
I20260620 14:20:31.887558 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4ea68561e14bf0bcfba08cedfbdfc7" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0ca64ef454dceacd54070fb218bd0" is_pre_election: true
I20260620 14:20:31.887658 2279 raft_consensus.cc:2454] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 0.
I20260620 14:20:31.887681 2559 raft_consensus.cc:2454] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 0.
I20260620 14:20:31.887923 2350 leader_election.cc:302] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a325f95b734d4a128a4ab18d88c140ce, a58b25134e6448b4b6b26eed83bc2bbe; no voters:
I20260620 14:20:31.888121 2711 raft_consensus.cc:2781] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:31.888180 2711 raft_consensus.cc:482] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:31.888208 2711 raft_consensus.cc:3037] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:31.889102 2711 raft_consensus.cc:504] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.889422 2711 leader_election.cc:288] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 election: Requested vote from peers a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787), 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679)
I20260620 14:20:31.889675 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4ea68561e14bf0bcfba08cedfbdfc7" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58b25134e6448b4b6b26eed83bc2bbe"
I20260620 14:20:31.889696 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4ea68561e14bf0bcfba08cedfbdfc7" candidate_uuid: "a325f95b734d4a128a4ab18d88c140ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0ca64ef454dceacd54070fb218bd0"
I20260620 14:20:31.889778 2279 raft_consensus.cc:3037] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:31.889822 2559 raft_consensus.cc:3037] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:31.890590 2279 raft_consensus.cc:2454] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 1.
I20260620 14:20:31.890594 2559 raft_consensus.cc:2454] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a325f95b734d4a128a4ab18d88c140ce in term 1.
I20260620 14:20:31.890846 2350 leader_election.cc:302] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a325f95b734d4a128a4ab18d88c140ce, a58b25134e6448b4b6b26eed83bc2bbe; no voters:
I20260620 14:20:31.891045 2711 raft_consensus.cc:2781] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:31.891150 2711 raft_consensus.cc:686] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 1 LEADER]: Becoming Leader. State: Replica: a325f95b734d4a128a4ab18d88c140ce, State: Running, Role: LEADER
I20260620 14:20:31.891276 2711 consensus_queue.cc:237] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260620 14:20:31.892580 2121 catalog_manager.cc:5495] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce reported cstate change: term changed from 0 to 1, leader changed from <none> to a325f95b734d4a128a4ab18d88c140ce (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a325f95b734d4a128a4ab18d88c140ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a325f95b734d4a128a4ab18d88c140ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: UNKNOWN } } }
I20260620 14:20:31.906872 2279 raft_consensus.cc:1261] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Refusing update from remote peer a325f95b734d4a128a4ab18d88c140ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:31.907045 2559 raft_consensus.cc:1261] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Refusing update from remote peer a325f95b734d4a128a4ab18d88c140ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:31.907291 2714 consensus_queue.cc:1035] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a58b25134e6448b4b6b26eed83bc2bbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:31.907455 2711 consensus_queue.cc:1035] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db0ca64ef454dceacd54070fb218bd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:20:31.919625 2703 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:20:33.466296 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:33.466464 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:33.468743 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 292985ee8b834703996a043366dd992f: Sending DeleteTablet for 3 replicas of tablet bf4ea68561e14bf0bcfba08cedfbdfc7
I20260620 14:20:33.469306 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet bf4ea68561e14bf0bcfba08cedfbdfc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39244
I20260620 14:20:33.469319 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet bf4ea68561e14bf0bcfba08cedfbdfc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39022
I20260620 14:20:33.469653 2773 tablet_replica.cc:330] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe: stopping tablet replica
I20260620 14:20:33.469638 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet bf4ea68561e14bf0bcfba08cedfbdfc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260620 14:20:33.469794 2773 raft_consensus.cc:2229] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a58b25134e6448b4b6b26eed83bc2bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:33.469838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 2194
I20260620 14:20:33.471037 2775 tablet_replica.cc:330] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: stopping tablet replica
I20260620 14:20:33.471189 2775 raft_consensus.cc:2229] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:33.471303 2775 raft_consensus.cc:2258] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:33.472021 2775 ts_tablet_manager.cc:1905] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:33.475515 2775 ts_tablet_manager.cc:1918] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:20:33.475636 2775 log.cc:1196] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/bf4ea68561e14bf0bcfba08cedfbdfc7
I20260620 14:20:33.475899 2774 tablet_replica.cc:330] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce: stopping tablet replica
I20260620 14:20:33.475934 2775 ts_tablet_manager.cc:1939] T bf4ea68561e14bf0bcfba08cedfbdfc7 P 4db0ca64ef454dceacd54070fb218bd0: Deleting consensus metadata
I20260620 14:20:33.476075 2774 raft_consensus.cc:2229] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:33.476307 2774 raft_consensus.cc:2258] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:33.476728 2109 catalog_manager.cc:4841] TS 4db0ca64ef454dceacd54070fb218bd0 (127.0.98.3:33679): tablet bf4ea68561e14bf0bcfba08cedfbdfc7 (table test-table [id=b6de389ce94a4d1d840a348791694996]) successfully deleted
I20260620 14:20:33.476939 2774 ts_tablet_manager.cc:1905] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:33.479032 2774 ts_tablet_manager.cc:1918] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:20:33.479140 2774 log.cc:1196] T bf4ea68561e14bf0bcfba08cedfbdfc7 P a325f95b734d4a128a4ab18d88c140ce: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/bf4ea68561e14bf0bcfba08cedfbdfc7
W20260620 14:20:33.479650 2106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32787 (error 108)
I20260620 14:20:33.479930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 2334
I20260620 14:20:33.481127 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:33.480730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32787 (local address 127.0.0.1:39268)
0620 14:20:33.480967 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:33.481043 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
W20260620 14:20:33.481397 2106 catalog_manager.cc:4584] TS a58b25134e6448b4b6b26eed83bc2bbe (127.0.98.1:32787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf4ea68561e14bf0bcfba08cedfbdfc7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32787: connect: Connection refused (error 111)
I20260620 14:20:33.488158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 2474
I20260620 14:20:33.489068 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:33.488280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43971 (local address 127.0.0.1:39058)
0620 14:20:33.488904 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:33.488966 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526}
I20260620 14:20:33.494836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 2090
2026-06-20T14:20:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6513 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:33 dist-test-slave-87nc krb5kdc[2780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:33 dist-test-slave-87nc krb5kdc[2780](info): set up 2 sockets
Jun 20 14:20:33 dist-test-slave-87nc krb5kdc[2780](info): commencing operation
krb5kdc: starting...
W20260620 14:20:34.535562 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:34 dist-test-slave-87nc krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:34.654812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45941
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:34.764107 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:34.764417 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:34.764495 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:34.768602 2793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:34.768695 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:34.768733 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:34.768790 2793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:34.768827 2793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:34.774232 2793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:34.775362 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:34.776482 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:34.785277 2798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:34.785305 2801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:34.785272 2799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:34.785353 2793 server_base.cc:1034] running on GCE node
I20260620 14:20:34.786136 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:34.787195 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:34.788384 2793 hybrid_clock.cc:648] HybridClock initialized: now 1781965234788356 us; error 51 us; skew 500 ppm
Jun 20 14:20:34 dist-test-slave-87nc krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:34.791006 2793 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:34.791909 2793 webserver.cc:460] Webserver started at http://127.0.98.62:41435/ using document root <none> and password file <none>
I20260620 14:20:34.792464 2793 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:34.792558 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:34.792789 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:34.794824 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "b09e75958472477a9fa7a349ea7411df"
format_stamp: "Formatted at 2026-06-20 14:20:34 on dist-test-slave-87nc"
server_key: "36213f21cc36530a3853f71404816f9a"
server_key_iv: "ebb16a652cad44d25f9008cb4b771602"
server_key_version: "encryptionkey@0"
I20260620 14:20:34.795404 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "b09e75958472477a9fa7a349ea7411df"
format_stamp: "Formatted at 2026-06-20 14:20:34 on dist-test-slave-87nc"
server_key: "36213f21cc36530a3853f71404816f9a"
server_key_iv: "ebb16a652cad44d25f9008cb4b771602"
server_key_version: "encryptionkey@0"
I20260620 14:20:34.799114 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 14:20:34.801556 2808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:34.802875 2793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:34.803031 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "b09e75958472477a9fa7a349ea7411df"
format_stamp: "Formatted at 2026-06-20 14:20:34 on dist-test-slave-87nc"
server_key: "36213f21cc36530a3853f71404816f9a"
server_key_iv: "ebb16a652cad44d25f9008cb4b771602"
server_key_version: "encryptionkey@0"
I20260620 14:20:34.803153 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:34.851502 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:34.854591 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:34.854820 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:34.861776 2793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45941
I20260620 14:20:34.861793 2860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45941 every 8 connection(s)
I20260620 14:20:34.862754 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:34.865593 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:34.871098 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df: Bootstrap starting.
I20260620 14:20:34.871737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 2793
I20260620 14:20:34.871892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:34.872148 392 external_mini_cluster.cc:1428] Setting key 1c0b150be61c79201279dd3e2eab45b0
I20260620 14:20:34.873946 2861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:34.874720 2861 log.cc:824] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:34.876966 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df: No bootstrap required, opened a new log
Jun 20 14:20:34 dist-test-slave-87nc krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:34.881670 2861 raft_consensus.cc:348] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09e75958472477a9fa7a349ea7411df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } }
I20260620 14:20:34.881923 2861 raft_consensus.cc:374] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:34.882023 2861 raft_consensus.cc:729] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09e75958472477a9fa7a349ea7411df, State: Initialized, Role: FOLLOWER
I20260620 14:20:34.882531 2861 consensus_queue.cc:260] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09e75958472477a9fa7a349ea7411df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } }
I20260620 14:20:34.882687 2861 raft_consensus.cc:388] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:34.882784 2861 raft_consensus.cc:482] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:34.882882 2861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:34.884150 2861 raft_consensus.cc:504] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09e75958472477a9fa7a349ea7411df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } }
I20260620 14:20:34.884538 2861 leader_election.cc:302] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b09e75958472477a9fa7a349ea7411df; no voters:
I20260620 14:20:34.884908 2861 leader_election.cc:288] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:34.884980 2866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:34.885186 2866 raft_consensus.cc:686] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [term 1 LEADER]: Becoming Leader. State: Replica: b09e75958472477a9fa7a349ea7411df, State: Running, Role: LEADER
I20260620 14:20:34.885548 2866 consensus_queue.cc:237] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09e75958472477a9fa7a349ea7411df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } }
I20260620 14:20:34.886258 2861 sys_catalog.cc:564] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:34.887373 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b09e75958472477a9fa7a349ea7411df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09e75958472477a9fa7a349ea7411df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } } }
I20260620 14:20:34.887404 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09e75958472477a9fa7a349ea7411df. Latest consensus state: current_term: 1 leader_uuid: "b09e75958472477a9fa7a349ea7411df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09e75958472477a9fa7a349ea7411df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } } }
I20260620 14:20:34.887601 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:34.887738 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:34.873396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.0.98.62:45941)
0620 14:20:34.873855 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:34.873870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:34.873906 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:20:34.874607 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:34.874626 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:34.875022 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:34.875418 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:34.876372 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:34.877404 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:34.878592 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:34.879051 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:34.879163 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:34.882039 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:34.882044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:34.882057 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:34.882152 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:34.884316 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:34.884779 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:34.884801 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:34.884807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:34.884939 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:34.885319 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:34.885322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:34.885324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:34.885801 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:34.886012 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:34.886324 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:20:34.886646 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":149,"threads_started":1}
I20260620 14:20:34.887594 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df [sys.catalog]: This master's current role is: LEADER
W20260620 14:20:34.891328 2881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:20:34.891399 2881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:20:34.891482 2875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:34.892282 2875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:34.898052 2875 catalog_manager.cc:1347] Generated new cluster ID: f1ed930d951945ddb4817c006a41a24c
I20260620 14:20:34.898125 2875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:34.910794 2875 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:34.912104 2875 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:34.917819 2875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b09e75958472477a9fa7a349ea7411df: Generated new TSK 0
I20260620 14:20:34.918627 2875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:20:34.954875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:35.073279 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:35.073568 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:35.073645 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:35.077826 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:35.077903 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:35.078007 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:35.083544 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:35.084718 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:35.085798 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:35.096112 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:35.096047 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:35.096094 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:35.096347 2887 server_base.cc:1034] running on GCE node
I20260620 14:20:35.096743 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:35.097409 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:35.098594 2887 hybrid_clock.cc:648] HybridClock initialized: now 1781965235098580 us; error 43 us; skew 500 ppm
Jun 20 14:20:35 dist-test-slave-87nc krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:35.101452 2887 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:35.102348 2887 webserver.cc:460] Webserver started at http://127.0.98.1:45309/ using document root <none> and password file <none>
I20260620 14:20:35.102912 2887 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:35.103005 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:35.103243 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:35.105295 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "945bdc1ef97841f982edcbec846e4723"
format_stamp: "Formatted at 2026-06-20 14:20:35 on dist-test-slave-87nc"
server_key: "88f86d73954dd47ff68f0c7a4c3e6552"
server_key_iv: "55096fb926334a799782c6638693e0b6"
server_key_version: "encryptionkey@0"
I20260620 14:20:35.105867 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "945bdc1ef97841f982edcbec846e4723"
format_stamp: "Formatted at 2026-06-20 14:20:35 on dist-test-slave-87nc"
server_key: "88f86d73954dd47ff68f0c7a4c3e6552"
server_key_iv: "55096fb926334a799782c6638693e0b6"
server_key_version: "encryptionkey@0"
I20260620 14:20:35.109809 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 14:20:35.112257 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:35.113474 2887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:35.113613 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "945bdc1ef97841f982edcbec846e4723"
format_stamp: "Formatted at 2026-06-20 14:20:35 on dist-test-slave-87nc"
server_key: "88f86d73954dd47ff68f0c7a4c3e6552"
server_key_iv: "55096fb926334a799782c6638693e0b6"
server_key_version: "encryptionkey@0"
I20260620 14:20:35.113729 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:35.130787 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:35.133695 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:35.133904 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:35.134509 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:35.135341 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:35.135413 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:35.135484 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:35.135532 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:35.144604 2887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42313
I20260620 14:20:35.144654 3015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42313 every 8 connection(s)
I20260620 14:20:35.145519 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:35.151463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 2887
I20260620 14:20:35.151599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:35.151904 392 external_mini_cluster.cc:1428] Setting key a2d24759bf67fe55dca5265066144f78
Jun 20 14:20:35 dist-test-slave-87nc krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:35.159550 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:35.147277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41479 (local address 127.0.98.62:45941)
0620 14:20:35.147432 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:35.147436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:35.148322 (+ 886us) server_negotiation.cc:407] Connection header received
0620 14:20:35.149663 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:35.149666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:35.149712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:35.149787 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:35.150875 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.151586 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.153003 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.153325 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.153448 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:35.155945 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:35.155949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:35.155952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:35.155982 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:35.157474 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:35.158025 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:35.158031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:35.158033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:35.158110 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:35.158457 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:35.158460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:35.158462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:35.158645 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:35.158751 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:35.159246 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 14:20:35.159374 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:20:35.160421 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:35.147677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45941 (local address 127.0.98.1:41479)
0620 14:20:35.148170 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:35.148213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:35.149440 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:35.149919 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:35.149926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:35.150278 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:35.150747 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:35.150760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.151727 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.151736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:35.152870 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:35.152880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.153481 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.153484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:35.153671 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:35.154291 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:35.154308 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:35.155734 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:35.157602 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:35.157607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:35.157621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:35.157889 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:35.158204 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:35.158207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:35.158209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:35.158352 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:35.158748 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:35.158755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:35.159006 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 14:20:35.159193 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 14:20:35.159530 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":129,"threads_started":1}
I20260620 14:20:35.161664 3016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45941
I20260620 14:20:35.161916 3016 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:35.162505 3016 heartbeater.cc:511] Master 127.0.98.62:45941 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:20:35.164103 2825 ts_manager.cc:194] Registered new tserver with Master: 945bdc1ef97841f982edcbec846e4723 (127.0.98.1:42313)
I20260620 14:20:35.165625 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:20:35.178151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:35.288455 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:35.288755 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:35.288836 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:35.293152 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:35.293226 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:35.293371 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:35.299472 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:35.300616 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:35.301728 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:35.311864 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:35.311982 3021 server_base.cc:1034] running on GCE node
W20260620 14:20:35.311879 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:35.311880 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:35.312547 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:35.313253 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:35.314443 3021 hybrid_clock.cc:648] HybridClock initialized: now 1781965235314432 us; error 25 us; skew 500 ppm
Jun 20 14:20:35 dist-test-slave-87nc krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:35.318423 3021 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:35.319275 3021 webserver.cc:460] Webserver started at http://127.0.98.2:42495/ using document root <none> and password file <none>
I20260620 14:20:35.319943 3021 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:35.320053 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:35.320264 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:35.323115 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "397a092aaf5540bcb48fedb2e0cbaa1f"
format_stamp: "Formatted at 2026-06-20 14:20:35 on dist-test-slave-87nc"
server_key: "fc1b1d6f87094f8f4b32998469a1f528"
server_key_iv: "5c31f46f864bd3f6ec7e726edb1dbae2"
server_key_version: "encryptionkey@0"
I20260620 14:20:35.323928 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "397a092aaf5540bcb48fedb2e0cbaa1f"
format_stamp: "Formatted at 2026-06-20 14:20:35 on dist-test-slave-87nc"
server_key: "fc1b1d6f87094f8f4b32998469a1f528"
server_key_iv: "5c31f46f864bd3f6ec7e726edb1dbae2"
server_key_version: "encryptionkey@0"
I20260620 14:20:35.328683 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:20:35.331993 3036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:35.333267 3021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:20:35.333396 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "397a092aaf5540bcb48fedb2e0cbaa1f"
format_stamp: "Formatted at 2026-06-20 14:20:35 on dist-test-slave-87nc"
server_key: "fc1b1d6f87094f8f4b32998469a1f528"
server_key_iv: "5c31f46f864bd3f6ec7e726edb1dbae2"
server_key_version: "encryptionkey@0"
I20260620 14:20:35.333539 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:35.354249 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:35.358613 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:35.358851 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:35.359681 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:35.360941 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:35.361024 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:35.361101 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:35.361141 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:35.370186 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39745
I20260620 14:20:35.370265 3149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39745 every 8 connection(s)
I20260620 14:20:35.371129 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:35.374364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 3021
I20260620 14:20:35.374452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:35.374709 392 external_mini_cluster.cc:1428] Setting key d6313745ad2365a56118b3ae438bdf02
Jun 20 14:20:35 dist-test-slave-87nc krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:20:35.385900 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:35.372909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57287 (local address 127.0.98.62:45941)
0620 14:20:35.373068 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:35.373073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:35.373877 (+ 804us) server_negotiation.cc:407] Connection header received
0620 14:20:35.375306 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:35.375309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:35.375363 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:35.375466 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:35.376684 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.377198 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.378954 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.379177 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.379262 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:35.382076 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:35.382080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:35.382084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:35.382112 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:35.383554 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:35.384364 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:35.384370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:35.384373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:35.384455 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:35.384817 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:35.384820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:35.384822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:35.385001 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:35.385117 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:35.385613 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 14:20:35.385731 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:20:35.386823 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:35.373314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45941 (local address 127.0.98.2:57287)
0620 14:20:35.373732 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:35.373771 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:35.375107 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:35.375653 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:35.375659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:35.376064 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:35.376548 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:35.376561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.377323 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.377331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:35.378828 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:35.378841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.379327 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.379334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:35.379574 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:35.380232 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:35.380255 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:35.381905 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:35.383675 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:35.383680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:35.383696 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:35.384260 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:35.384570 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:35.384573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:35.384575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:35.384715 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:35.385115 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:35.385122 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:35.385370 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 14:20:35.385561 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:20:35.385875 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":121,"threads_started":1}
I20260620 14:20:35.388226 3150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45941
I20260620 14:20:35.388461 3150 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:35.389016 3150 heartbeater.cc:511] Master 127.0.98.62:45941 requested a full tablet report, sending...
I20260620 14:20:35.390108 2825 ts_manager.cc:194] Registered new tserver with Master: 397a092aaf5540bcb48fedb2e0cbaa1f (127.0.98.2:39745)
I20260620 14:20:35.390950 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:20:35.402109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:35.512333 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:35.512598 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:35.512671 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:35.516834 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:35.516902 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:35.517012 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:35.522632 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:35.523795 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:35.524842 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:35.534327 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:35.534402 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:35.534328 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:35.534559 3155 server_base.cc:1034] running on GCE node
I20260620 14:20:35.535072 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:35.535701 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:35.536870 3155 hybrid_clock.cc:648] HybridClock initialized: now 1781965235536832 us; error 49 us; skew 500 ppm
Jun 20 14:20:35 dist-test-slave-87nc krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:35.539628 3155 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:35.540520 3155 webserver.cc:460] Webserver started at http://127.0.98.3:36755/ using document root <none> and password file <none>
I20260620 14:20:35.541078 3155 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:35.541169 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:35.541392 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:35.543409 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "fc8b0dc6db7343e2b34951974a425dd6"
format_stamp: "Formatted at 2026-06-20 14:20:35 on dist-test-slave-87nc"
server_key: "4ad502c9729cd59e6a67b66bcda73d14"
server_key_iv: "cfeeb90a18c4658de6842f25fddfa672"
server_key_version: "encryptionkey@0"
I20260620 14:20:35.543996 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fc8b0dc6db7343e2b34951974a425dd6"
format_stamp: "Formatted at 2026-06-20 14:20:35 on dist-test-slave-87nc"
server_key: "4ad502c9729cd59e6a67b66bcda73d14"
server_key_iv: "cfeeb90a18c4658de6842f25fddfa672"
server_key_version: "encryptionkey@0"
I20260620 14:20:35.547560 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:20:35.549942 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:35.551113 3155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:35.551250 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "fc8b0dc6db7343e2b34951974a425dd6"
format_stamp: "Formatted at 2026-06-20 14:20:35 on dist-test-slave-87nc"
server_key: "4ad502c9729cd59e6a67b66bcda73d14"
server_key_iv: "cfeeb90a18c4658de6842f25fddfa672"
server_key_version: "encryptionkey@0"
I20260620 14:20:35.551363 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:35.578461 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:35.581457 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:35.581670 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:35.582273 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:35.583099 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:35.583174 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:35.583246 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:35.583290 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:35.592404 3155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34517
I20260620 14:20:35.592425 3283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34517 every 8 connection(s)
I20260620 14:20:35.593307 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:35.598726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 3155
I20260620 14:20:35.598847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:35.599152 392 external_mini_cluster.cc:1428] Setting key 60ff28e358b6ffb4404d9c41e78d173e
Jun 20 14:20:35 dist-test-slave-87nc krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:35.606943 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:35.595055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45795 (local address 127.0.98.62:45941)
0620 14:20:35.595210 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:35.595214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:35.596105 (+ 891us) server_negotiation.cc:407] Connection header received
0620 14:20:35.597395 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:35.597398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:35.597444 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:35.597521 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:35.598673 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.599444 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.601032 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.601348 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.601429 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:35.603847 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:35.603850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:35.603852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:35.603876 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:35.605066 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:35.605507 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:35.605519 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:35.605520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:35.605592 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:35.605864 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:35.605866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:35.605867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:35.606050 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:35.606125 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:35.606673 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 14:20:35.606763 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:20:35.607717 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:35.595471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45941 (local address 127.0.98.3:45795)
0620 14:20:35.595953 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:35.596001 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:35.597204 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:35.597687 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:35.597693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:35.598047 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:35.598531 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:35.598543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.599587 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.599593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:35.600915 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:35.600926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:35.601483 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:35.601486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:35.601623 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:35.602229 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:35.602245 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:35.603652 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:35.605164 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:35.605168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:35.605180 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:35.605419 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:35.605670 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:35.605672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:35.605673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:35.605793 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:35.606132 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:35.606137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:35.606419 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 14:20:35.606580 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 14:20:35.606812 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260620 14:20:35.608835 3284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45941
I20260620 14:20:35.609043 3284 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:35.609455 3284 heartbeater.cc:511] Master 127.0.98.62:45941 requested a full tablet report, sending...
I20260620 14:20:35.610355 2825 ts_manager.cc:194] Registered new tserver with Master: fc8b0dc6db7343e2b34951974a425dd6 (127.0.98.3:34517)
I20260620 14:20:35.610845 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45795
I20260620 14:20:35.612869 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:20:35.625525 2864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:35.624044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.98.62:45941)
0620 14:20:35.624176 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:35.624182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:35.624428 (+ 246us) server_negotiation.cc:407] Connection header received
0620 14:20:35.624567 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:35.624570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:35.624615 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:35.624709 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:35.625381 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38262: BlockingRecv error: recv got EOF from 127.0.0.1:38262 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:20:35.626490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 2887
I20260620 14:20:35.632360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 3021
I20260620 14:20:35.637877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 3155
I20260620 14:20:35.642937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 2793
2026-06-20T14:20:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2146 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:35 dist-test-slave-87nc krb5kdc[3296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:35 dist-test-slave-87nc krb5kdc[3296](info): set up 2 sockets
Jun 20 14:20:35 dist-test-slave-87nc krb5kdc[3296](info): commencing operation
krb5kdc: starting...
W20260620 14:20:36.679708 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:36 dist-test-slave-87nc krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:36.798152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:36.905050 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:36.905294 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:36.905349 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:36.909303 3309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:36.909358 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:36.909375 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:36.909391 3309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:36.909405 3309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:36.914513 3309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:36.915424 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:36.916437 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:36.924408 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:36.924489 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:36.924507 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:36.924598 3309 server_base.cc:1034] running on GCE node
I20260620 14:20:36.925179 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:36.926221 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:36.927381 3309 hybrid_clock.cc:648] HybridClock initialized: now 1781965236927361 us; error 31 us; skew 500 ppm
Jun 20 14:20:36 dist-test-slave-87nc krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:36.933568 3309 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:36.934507 3309 webserver.cc:460] Webserver started at http://127.0.98.62:39193/ using document root <none> and password file <none>
I20260620 14:20:36.935077 3309 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:36.935139 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:36.935386 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:36.937677 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "7ab223a97857448288316853a517100b"
format_stamp: "Formatted at 2026-06-20 14:20:36 on dist-test-slave-87nc"
server_key: "4bf99f753d9a5acc797397ffa0b53844"
server_key_iv: "a03de5657d85d8dbff30ac93c9a301db"
server_key_version: "encryptionkey@0"
I20260620 14:20:36.938424 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ab223a97857448288316853a517100b"
format_stamp: "Formatted at 2026-06-20 14:20:36 on dist-test-slave-87nc"
server_key: "4bf99f753d9a5acc797397ffa0b53844"
server_key_iv: "a03de5657d85d8dbff30ac93c9a301db"
server_key_version: "encryptionkey@0"
I20260620 14:20:36.943742 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 14:20:36.946662 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:36.948055 3309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:20:36.948180 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "7ab223a97857448288316853a517100b"
format_stamp: "Formatted at 2026-06-20 14:20:36 on dist-test-slave-87nc"
server_key: "4bf99f753d9a5acc797397ffa0b53844"
server_key_iv: "a03de5657d85d8dbff30ac93c9a301db"
server_key_version: "encryptionkey@0"
I20260620 14:20:36.948283 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:36.963675 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:36.966565 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:36.966758 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:36.973654 3309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46517
I20260620 14:20:36.973927 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46517 every 8 connection(s)
I20260620 14:20:36.974660 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:36.977203 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:36.982311 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b: Bootstrap starting.
I20260620 14:20:36.983888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 3309
I20260620 14:20:36.984012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:36.984284 392 external_mini_cluster.cc:1428] Setting key 61d3b55f17b070e65359bdd58a9f126e
I20260620 14:20:36.984882 3377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:36.985757 3377 log.cc:824] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:36.987506 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b: No bootstrap required, opened a new log
I20260620 14:20:36.990677 3377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab223a97857448288316853a517100b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260620 14:20:36.990949 3377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:36.991040 3377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab223a97857448288316853a517100b, State: Initialized, Role: FOLLOWER
I20260620 14:20:36.991451 3377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab223a97857448288316853a517100b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260620 14:20:36.991621 3377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:36.991700 3377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:36.991832 3377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:20:36 dist-test-slave-87nc krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:36.992919 3377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab223a97857448288316853a517100b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260620 14:20:36.993279 3377 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ab223a97857448288316853a517100b; no voters:
I20260620 14:20:36.993630 3377 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:36.993729 3382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:36.993976 3382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [term 1 LEADER]: Becoming Leader. State: Replica: 7ab223a97857448288316853a517100b, State: Running, Role: LEADER
I20260620 14:20:36.994355 3382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab223a97857448288316853a517100b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260620 14:20:36.994962 3377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:36.996078 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ab223a97857448288316853a517100b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab223a97857448288316853a517100b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } }
I20260620 14:20:36.996069 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ab223a97857448288316853a517100b. Latest consensus state: current_term: 1 leader_uuid: "7ab223a97857448288316853a517100b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab223a97857448288316853a517100b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } }
I20260620 14:20:36.996207 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:36.996207 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:36.996675 3391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:36.999066 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:36.985285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.0.98.62:46517)
0620 14:20:36.985720 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:36.985733 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:36.985765 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:20:36.986594 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:36.986613 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:36.986924 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:36.987259 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:36.988098 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:36.989010 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:36.989852 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:36.990310 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:36.990483 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:36.993535 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:36.993545 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:36.993566 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:36.993600 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:36.995869 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:36.996406 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:36.996412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:36.996418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:36.996567 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:36.996908 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:36.996911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:36.996913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:36.997499 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:36.997692 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:36.998097 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 14:20:36.998396 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260620 14:20:36.999689 3391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:37.002035 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:20:37.005271 3391 catalog_manager.cc:1347] Generated new cluster ID: c528737b87aa42569597602d7a0c1ada
I20260620 14:20:37.005347 3391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:37.016932 3391 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:37.018087 3391 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:37.024497 3391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ab223a97857448288316853a517100b: Generated new TSK 0
I20260620 14:20:37.024673 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:37.019710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.0.98.62:46517)
0620 14:20:37.019849 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:37.019854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:37.020097 (+ 243us) server_negotiation.cc:407] Connection header received
0620 14:20:37.020253 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:37.020257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:37.020305 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:37.020386 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:37.020873 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:37.021394 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:37.022232 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:37.022467 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:37.022573 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:37.023496 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:37.023500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:37.023504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:37.024101 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:37.024135 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:37.024154 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:37.024241 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:37.024396 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:20:37.024457 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:20:37.025329 3391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:20:37.033515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 3309
2026-06-20T14:20:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1388 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:37 dist-test-slave-87nc krb5kdc[3410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:37 dist-test-slave-87nc krb5kdc[3410](info): set up 2 sockets
Jun 20 14:20:37 dist-test-slave-87nc krb5kdc[3410](info): commencing operation
krb5kdc: starting...
W20260620 14:20:38.107872 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:38 dist-test-slave-87nc krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:38.224747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:38.330368 3423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:38.330657 3423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:38.330724 3423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:38.335076 3423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:38.335161 3423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:38.335201 3423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:38.335237 3423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:38.335274 3423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:38.340533 3423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:38.341534 3423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:38.342543 3423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:38.350685 3428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:38.350689 3429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:38.350708 3431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:38.351027 3423 server_base.cc:1034] running on GCE node
I20260620 14:20:38.351648 3423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:38.352689 3423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:38.353874 3423 hybrid_clock.cc:648] HybridClock initialized: now 1781965238353863 us; error 24 us; skew 500 ppm
Jun 20 14:20:38 dist-test-slave-87nc krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:38.356544 3423 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:38.357374 3423 webserver.cc:460] Webserver started at http://127.0.98.62:41491/ using document root <none> and password file <none>
I20260620 14:20:38.357860 3423 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:38.357918 3423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:38.358086 3423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:38.360081 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "b0e780d9bd4e443089c7d0fdcc3cd650"
format_stamp: "Formatted at 2026-06-20 14:20:38 on dist-test-slave-87nc"
server_key: "9f3efde81579e3f24956bb5bd8fe4b25"
server_key_iv: "4841e871c813a870f7bfa71c87191df0"
server_key_version: "encryptionkey@0"
I20260620 14:20:38.360575 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0e780d9bd4e443089c7d0fdcc3cd650"
format_stamp: "Formatted at 2026-06-20 14:20:38 on dist-test-slave-87nc"
server_key: "9f3efde81579e3f24956bb5bd8fe4b25"
server_key_iv: "4841e871c813a870f7bfa71c87191df0"
server_key_version: "encryptionkey@0"
I20260620 14:20:38.363992 3423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:20:38.366212 3438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:38.367442 3423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:38.367564 3423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "b0e780d9bd4e443089c7d0fdcc3cd650"
format_stamp: "Formatted at 2026-06-20 14:20:38 on dist-test-slave-87nc"
server_key: "9f3efde81579e3f24956bb5bd8fe4b25"
server_key_iv: "4841e871c813a870f7bfa71c87191df0"
server_key_version: "encryptionkey@0"
I20260620 14:20:38.367667 3423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:38.378247 3423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:38.381430 3423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:38.381628 3423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:38.388443 3423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43065
I20260620 14:20:38.388731 3490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43065 every 8 connection(s)
I20260620 14:20:38.389390 3423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:38.390497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 3423
I20260620 14:20:38.390589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:38.390867 392 external_mini_cluster.cc:1428] Setting key b514d7c23f53c9d8637c9171f2d4610f
I20260620 14:20:38.392479 3491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:38.397924 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650: Bootstrap starting.
Jun 20 14:20:38 dist-test-slave-87nc krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:38.400461 3491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:38.401160 3491 log.cc:824] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:38.403231 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650: No bootstrap required, opened a new log
I20260620 14:20:38.404965 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:38.392204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.0.98.62:43065)
0620 14:20:38.392640 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:38.392654 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:38.392688 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:20:38.393307 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:38.393323 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:38.393635 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:38.393971 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:38.394707 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.395649 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.396605 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.397153 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.397310 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:38.400122 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:38.400127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:38.400138 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:38.400163 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:38.402054 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:38.402516 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:38.402520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:38.402524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:38.402620 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:38.402901 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:38.402902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:38.402903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:38.403256 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:38.403433 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:38.403803 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 14:20:38.404108 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260620 14:20:38.406320 3491 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e780d9bd4e443089c7d0fdcc3cd650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } }
I20260620 14:20:38.406538 3491 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:38.406599 3491 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e780d9bd4e443089c7d0fdcc3cd650, State: Initialized, Role: FOLLOWER
I20260620 14:20:38.407068 3491 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e780d9bd4e443089c7d0fdcc3cd650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } }
I20260620 14:20:38.407233 3491 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:38.407301 3491 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:38.407395 3491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:38.408330 3491 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e780d9bd4e443089c7d0fdcc3cd650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } }
I20260620 14:20:38.408664 3491 leader_election.cc:302] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e780d9bd4e443089c7d0fdcc3cd650; no voters:
I20260620 14:20:38.409014 3491 leader_election.cc:288] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:38.409116 3496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:38.409322 3496 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [term 1 LEADER]: Becoming Leader. State: Replica: b0e780d9bd4e443089c7d0fdcc3cd650, State: Running, Role: LEADER
I20260620 14:20:38.409667 3496 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e780d9bd4e443089c7d0fdcc3cd650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } }
I20260620 14:20:38.410362 3491 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:38.410925 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e780d9bd4e443089c7d0fdcc3cd650. Latest consensus state: current_term: 1 leader_uuid: "b0e780d9bd4e443089c7d0fdcc3cd650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e780d9bd4e443089c7d0fdcc3cd650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } } }
I20260620 14:20:38.411093 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:38.411563 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:38.411564 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0e780d9bd4e443089c7d0fdcc3cd650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e780d9bd4e443089c7d0fdcc3cd650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } } }
I20260620 14:20:38.411731 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:38.414780 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:38.419683 3501 catalog_manager.cc:1347] Generated new cluster ID: 74738c9d52e7408e89feeafe24772cba
I20260620 14:20:38.420152 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:38.432241 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:38.433039 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:38.438625 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e780d9bd4e443089c7d0fdcc3cd650: Generated new TSK 0
I20260620 14:20:38.439318 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:20:38.482510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43065
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:38.594465 3517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:38.594722 3517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:38.594764 3517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:38.599009 3517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:38.599074 3517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:38.599174 3517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:38.604398 3517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:38.605420 3517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:38.606442 3517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:38.615638 3522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:38.615671 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:38.615715 3523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:38.615980 3517 server_base.cc:1034] running on GCE node
I20260620 14:20:38.616339 3517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:38.616935 3517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:38.618129 3517 hybrid_clock.cc:648] HybridClock initialized: now 1781965238618114 us; error 28 us; skew 500 ppm
Jun 20 14:20:38 dist-test-slave-87nc krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:38.620858 3517 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:38.621639 3517 webserver.cc:460] Webserver started at http://127.0.98.1:35327/ using document root <none> and password file <none>
I20260620 14:20:38.622146 3517 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:38.622205 3517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:38.622372 3517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:38.624394 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "40cae3652cde4c899d747cbe167a2ed3"
format_stamp: "Formatted at 2026-06-20 14:20:38 on dist-test-slave-87nc"
server_key: "aed9b2dc1b0a6d9ff84a569215299529"
server_key_iv: "41680da36511d932fffa29b520821b3f"
server_key_version: "encryptionkey@0"
I20260620 14:20:38.625020 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40cae3652cde4c899d747cbe167a2ed3"
format_stamp: "Formatted at 2026-06-20 14:20:38 on dist-test-slave-87nc"
server_key: "aed9b2dc1b0a6d9ff84a569215299529"
server_key_iv: "41680da36511d932fffa29b520821b3f"
server_key_version: "encryptionkey@0"
I20260620 14:20:38.630404 3517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:20:38.633113 3532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:38.634387 3517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:38.634480 3517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "40cae3652cde4c899d747cbe167a2ed3"
format_stamp: "Formatted at 2026-06-20 14:20:38 on dist-test-slave-87nc"
server_key: "aed9b2dc1b0a6d9ff84a569215299529"
server_key_iv: "41680da36511d932fffa29b520821b3f"
server_key_version: "encryptionkey@0"
I20260620 14:20:38.634569 3517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:38.661139 3517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:38.663941 3517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:38.664093 3517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:38.664623 3517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:38.665374 3517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:38.665421 3517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:38.665459 3517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:38.665478 3517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:38.674438 3517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45133
I20260620 14:20:38.674474 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45133 every 8 connection(s)
I20260620 14:20:38.675345 3517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:38.679883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 3517
I20260620 14:20:38.680007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:38.680363 392 external_mini_cluster.cc:1428] Setting key 84f398f6312047b5d2607cb83f03bf03
Jun 20 14:20:38 dist-test-slave-87nc krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:20:38.690654 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:38.677206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35815 (local address 127.0.98.62:43065)
0620 14:20:38.677339 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:38.677342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:38.678159 (+ 817us) server_negotiation.cc:407] Connection header received
0620 14:20:38.679910 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:38.679913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:38.679970 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:38.680065 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:38.681538 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.682159 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.683734 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.684173 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.684304 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:38.687074 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:38.687080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:38.687086 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:38.687121 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:38.688766 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:38.689241 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:38.689245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:38.689247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:38.689336 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:38.689651 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:38.689654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:38.689656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:38.689900 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:38.690021 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:38.690434 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 14:20:38.690535 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:20:38.691658 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:38.677578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43065 (local address 127.0.98.1:35815)
0620 14:20:38.678007 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:38.678050 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:38.679658 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:38.680281 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:38.680291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:38.680781 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:38.681346 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:38.681360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.682298 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.682305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:38.683608 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:38.683618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.684404 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.684408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:38.684544 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:38.685144 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:38.685159 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:38.686895 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:38.688876 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:38.688881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:38.688892 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:38.689135 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:38.689432 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:38.689434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:38.689435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:38.689561 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:38.690005 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:38.690010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:38.690227 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 14:20:38.690375 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 14:20:38.690620 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:20:38.692843 3646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43065
I20260620 14:20:38.693064 3646 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:38.693595 3646 heartbeater.cc:511] Master 127.0.98.62:43065 requested a full tablet report, sending...
I20260620 14:20:38.695281 3455 ts_manager.cc:194] Registered new tserver with Master: 40cae3652cde4c899d747cbe167a2ed3 (127.0.98.1:45133)
I20260620 14:20:38.696956 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:20:38.706809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43065
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:38.816561 3651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:38.816843 3651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:38.816926 3651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:38.821081 3651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:38.821144 3651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:38.821203 3651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:38.826562 3651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:38.827589 3651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:38.828647 3651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:38.838186 3657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:38.838330 3651 server_base.cc:1034] running on GCE node
W20260620 14:20:38.838142 3656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:38.838472 3659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:38.838833 3651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:38.839421 3651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:38.840603 3651 hybrid_clock.cc:648] HybridClock initialized: now 1781965238840581 us; error 34 us; skew 500 ppm
Jun 20 14:20:38 dist-test-slave-87nc krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:38.843334 3651 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:38.844189 3651 webserver.cc:460] Webserver started at http://127.0.98.2:37369/ using document root <none> and password file <none>
I20260620 14:20:38.844717 3651 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:38.844802 3651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:38.845029 3651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:38.846935 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "7d4456ff19044182bb72fea7dedcc6fc"
format_stamp: "Formatted at 2026-06-20 14:20:38 on dist-test-slave-87nc"
server_key: "c62ceddd1fa820f7c475e5ef6547bebe"
server_key_iv: "4e3728ee5b3f4d39c242ccb0004b5305"
server_key_version: "encryptionkey@0"
I20260620 14:20:38.847481 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7d4456ff19044182bb72fea7dedcc6fc"
format_stamp: "Formatted at 2026-06-20 14:20:38 on dist-test-slave-87nc"
server_key: "c62ceddd1fa820f7c475e5ef6547bebe"
server_key_iv: "4e3728ee5b3f4d39c242ccb0004b5305"
server_key_version: "encryptionkey@0"
I20260620 14:20:38.851025 3651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:20:38.853405 3666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:38.854550 3651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:38.854672 3651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "7d4456ff19044182bb72fea7dedcc6fc"
format_stamp: "Formatted at 2026-06-20 14:20:38 on dist-test-slave-87nc"
server_key: "c62ceddd1fa820f7c475e5ef6547bebe"
server_key_iv: "4e3728ee5b3f4d39c242ccb0004b5305"
server_key_version: "encryptionkey@0"
I20260620 14:20:38.854772 3651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:38.871594 3651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:38.874841 3651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:38.875046 3651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:38.875638 3651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:38.876504 3651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:38.876574 3651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:38.876646 3651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:38.876689 3651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:38.885509 3651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37169
I20260620 14:20:38.885533 3779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37169 every 8 connection(s)
I20260620 14:20:38.886418 3651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:38.893003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 3651
I20260620 14:20:38.893141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:38.893430 392 external_mini_cluster.cc:1428] Setting key ec06c7f735820addee5fcfc54f6d9494
Jun 20 14:20:38 dist-test-slave-87nc krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:38.900919 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:38.888236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58719 (local address 127.0.98.62:43065)
0620 14:20:38.888370 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:38.888373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:38.889209 (+ 836us) server_negotiation.cc:407] Connection header received
0620 14:20:38.890630 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:38.890634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:38.890688 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:38.890790 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:38.891989 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.892506 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.893788 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.894017 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.894131 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:38.896845 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:38.896850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:38.896855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:38.896889 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:38.898670 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:38.899188 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:38.899195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:38.899197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:38.899295 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:38.899644 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:38.899647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:38.899649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:38.899931 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:38.900053 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:38.900518 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 14:20:38.900671 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:20:38.901644 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:38.888640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43065 (local address 127.0.98.2:58719)
0620 14:20:38.889067 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:38.889106 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:38.890389 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:38.890961 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:38.890968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:38.891310 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:38.891842 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:38.891864 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.892628 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.892632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:38.893669 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:38.893678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:38.894163 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:38.894167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:38.894336 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:38.894942 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:38.894960 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:38.896678 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:38.898784 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:38.898789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:38.898801 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:38.899056 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:38.899403 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:38.899405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:38.899407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:38.899546 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:38.900043 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:38.900048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:38.900276 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 14:20:38.900431 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 14:20:38.900698 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260620 14:20:38.902823 3780 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43065
I20260620 14:20:38.903086 3780 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:20:38.903601 3780 heartbeater.cc:511] Master 127.0.98.62:43065 requested a full tablet report, sending...
I20260620 14:20:38.904675 3455 ts_manager.cc:194] Registered new tserver with Master: 7d4456ff19044182bb72fea7dedcc6fc (127.0.98.2:37169)
I20260620 14:20:38.905251 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:20:38.920601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43065
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:39.029584 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:39.029832 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:39.029872 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:39.033747 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:39.033793 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:39.033847 3785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:39.039243 3785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:39.040235 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:39.041169 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:39.050383 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:39.050431 3785 server_base.cc:1034] running on GCE node
W20260620 14:20:39.050491 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:39.050374 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:39.050997 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:39.051571 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:39.052719 3785 hybrid_clock.cc:648] HybridClock initialized: now 1781965239052675 us; error 56 us; skew 500 ppm
Jun 20 14:20:39 dist-test-slave-87nc krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:39.055248 3785 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:39.056133 3785 webserver.cc:460] Webserver started at http://127.0.98.3:37585/ using document root <none> and password file <none>
I20260620 14:20:39.056656 3785 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:39.056741 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:39.056960 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:39.058977 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "68a92d9afc164318a2a197580829d330"
format_stamp: "Formatted at 2026-06-20 14:20:39 on dist-test-slave-87nc"
server_key: "e748d603c357bc40c1aa267c5f153fc6"
server_key_iv: "112e8eab39347de5d6f1d094d3ee89a0"
server_key_version: "encryptionkey@0"
I20260620 14:20:39.059510 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68a92d9afc164318a2a197580829d330"
format_stamp: "Formatted at 2026-06-20 14:20:39 on dist-test-slave-87nc"
server_key: "e748d603c357bc40c1aa267c5f153fc6"
server_key_iv: "112e8eab39347de5d6f1d094d3ee89a0"
server_key_version: "encryptionkey@0"
I20260620 14:20:39.062937 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:20:39.065219 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:39.066396 3785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:39.066525 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "68a92d9afc164318a2a197580829d330"
format_stamp: "Formatted at 2026-06-20 14:20:39 on dist-test-slave-87nc"
server_key: "e748d603c357bc40c1aa267c5f153fc6"
server_key_iv: "112e8eab39347de5d6f1d094d3ee89a0"
server_key_version: "encryptionkey@0"
I20260620 14:20:39.066629 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:39.078339 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:39.081168 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:39.081369 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:39.081936 3785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:39.082732 3785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:39.082803 3785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:39.082872 3785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:39.082926 3785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:39.091827 3785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43805
I20260620 14:20:39.091874 3913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43805 every 8 connection(s)
I20260620 14:20:39.092736 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:39.096484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 3785
I20260620 14:20:39.096611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:39.096891 392 external_mini_cluster.cc:1428] Setting key cd62fc29e97d966aeb800c56753f15ec
Jun 20 14:20:39 dist-test-slave-87nc krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:39.106150 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:39.094528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40619 (local address 127.0.98.62:43065)
0620 14:20:39.094666 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:39.094670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:39.095497 (+ 827us) server_negotiation.cc:407] Connection header received
0620 14:20:39.097266 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:39.097270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:39.097331 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:39.097424 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:39.098873 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:39.099475 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:39.100614 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:39.100816 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:39.100891 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:39.103151 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:39.103154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:39.103156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:39.103180 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:39.104529 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:39.104944 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:39.104947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:39.104948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:39.105017 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:39.105281 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:39.105283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:39.105284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:39.105449 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:39.105531 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:39.105944 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 14:20:39.106035 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:20:39.106930 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:39.094923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43065 (local address 127.0.98.3:40619)
0620 14:20:39.095349 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:39.095390 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:39.096996 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:39.097609 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:39.097616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:39.098166 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:39.098726 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:39.098739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:39.099586 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:39.099589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:39.100510 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:39.100519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:39.100932 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:39.100935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:39.101075 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:39.101601 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:39.101616 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:39.103009 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:39.104625 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:39.104629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:39.104640 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:39.104864 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:39.105090 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:39.105092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:39.105093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:39.105207 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:39.105536 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:39.105541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:39.105742 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 14:20:39.105884 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 14:20:39.106135 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260620 14:20:39.108047 3914 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43065
I20260620 14:20:39.108248 3914 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:39.108656 3914 heartbeater.cc:511] Master 127.0.98.62:43065 requested a full tablet report, sending...
I20260620 14:20:39.109650 3455 ts_manager.cc:194] Registered new tserver with Master: 68a92d9afc164318a2a197580829d330 (127.0.98.3:43805)
I20260620 14:20:39.110365 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40619
I20260620 14:20:39.110769 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:20:39 dist-test-slave-87nc krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965239, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:20:39 dist-test-slave-87nc krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965239, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:39.134486 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:39.126815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.0.98.62:43065)
0620 14:20:39.126938 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:39.126942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:39.127058 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:20:39.127165 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:39.127167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:39.127208 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:39.127280 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:39.127967 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:39.128470 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:39.129246 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:39.129445 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:39.129521 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:39.131844 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:39.131847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:39.131849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:39.131870 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:39.133113 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:39.133465 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:39.133467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:39.133468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:39.133541 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:39.133751 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:39.133753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:39.133754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:39.133946 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:39.134056 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:39.134284 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:20:39.134371 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 14:20:39.134840 3455 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60928
I20260620 14:20:39.135627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 3517
I20260620 14:20:39.141258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 3651
I20260620 14:20:39.146528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 3785
I20260620 14:20:39.151587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 3423
2026-06-20T14:20:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2119 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:39 dist-test-slave-87nc krb5kdc[3926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:39 dist-test-slave-87nc krb5kdc[3926](info): set up 2 sockets
Jun 20 14:20:39 dist-test-slave-87nc krb5kdc[3926](info): commencing operation
krb5kdc: starting...
W20260620 14:20:40.179988 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:40 dist-test-slave-87nc krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:40.296339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:40.404765 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:40.405042 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:40.405081 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:40.409104 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:40.409165 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:40.409184 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:40.409200 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:40.409215 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:40.414561 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:40.415684 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:40.416783 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:40.425194 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:40.425182 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:40.425267 3939 server_base.cc:1034] running on GCE node
W20260620 14:20:40.425195 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:40.426110 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:40.427248 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:40.428445 3939 hybrid_clock.cc:648] HybridClock initialized: now 1781965240428441 us; error 35 us; skew 500 ppm
Jun 20 14:20:40 dist-test-slave-87nc krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:40.431316 3939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:40.432260 3939 webserver.cc:460] Webserver started at http://127.0.98.62:45715/ using document root <none> and password file <none>
I20260620 14:20:40.432775 3939 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:40.432832 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:40.432989 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:40.434918 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "6e6590a0bb9f462faf563903a1db0bf1"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-87nc"
server_key: "a280e33961d63d2cfabc75d1e6f594e6"
server_key_iv: "506c8f0622cab68ddf9b22da790d7da1"
server_key_version: "encryptionkey@0"
I20260620 14:20:40.435406 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e6590a0bb9f462faf563903a1db0bf1"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-87nc"
server_key: "a280e33961d63d2cfabc75d1e6f594e6"
server_key_iv: "506c8f0622cab68ddf9b22da790d7da1"
server_key_version: "encryptionkey@0"
I20260620 14:20:40.438952 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:20:40.441231 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:40.442451 3939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:40.442592 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "6e6590a0bb9f462faf563903a1db0bf1"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-87nc"
server_key: "a280e33961d63d2cfabc75d1e6f594e6"
server_key_iv: "506c8f0622cab68ddf9b22da790d7da1"
server_key_version: "encryptionkey@0"
I20260620 14:20:40.442723 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:40.457878 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:40.461354 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:40.461583 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:40.468508 3939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37391
I20260620 14:20:40.468562 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37391 every 8 connection(s)
I20260620 14:20:40.469537 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:40.472067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 3939
I20260620 14:20:40.472211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:40.472484 392 external_mini_cluster.cc:1428] Setting key 88aac9134bfc1706d0965ffbccdfbecc
I20260620 14:20:40.472541 4007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:40.480206 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1: Bootstrap starting.
Jun 20 14:20:40 dist-test-slave-87nc krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:40.482457 4007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:40.483389 4007 log.cc:824] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:40.485761 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1: No bootstrap required, opened a new log
I20260620 14:20:40.488171 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:40.473960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.0.98.62:37391)
0620 14:20:40.474390 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:40.474414 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:40.474457 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:20:40.475449 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:40.475473 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:40.475933 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:40.476323 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:40.477142 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.478604 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.479442 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.480131 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.480300 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:40.482770 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:40.482776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:40.482792 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:40.482825 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:40.484821 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:40.485323 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:40.485328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:40.485334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:40.485478 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:40.485785 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:40.485788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:40.485790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:40.486338 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:40.486529 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:40.486843 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 14:20:40.487132 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260620 14:20:40.490175 4007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6590a0bb9f462faf563903a1db0bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } }
I20260620 14:20:40.490461 4007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:40.490558 4007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6590a0bb9f462faf563903a1db0bf1, State: Initialized, Role: FOLLOWER
I20260620 14:20:40.491039 4007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6590a0bb9f462faf563903a1db0bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } }
I20260620 14:20:40.491184 4007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:40.491245 4007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:40.491340 4007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:40.492368 4007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6590a0bb9f462faf563903a1db0bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } }
I20260620 14:20:40.492707 4007 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e6590a0bb9f462faf563903a1db0bf1; no voters:
I20260620 14:20:40.493044 4007 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:40.493112 4012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:40.493305 4012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [term 1 LEADER]: Becoming Leader. State: Replica: 6e6590a0bb9f462faf563903a1db0bf1, State: Running, Role: LEADER
I20260620 14:20:40.493619 4012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6590a0bb9f462faf563903a1db0bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } }
I20260620 14:20:40.494294 4007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:40.495460 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e6590a0bb9f462faf563903a1db0bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6590a0bb9f462faf563903a1db0bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } } }
I20260620 14:20:40.495522 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6590a0bb9f462faf563903a1db0bf1. Latest consensus state: current_term: 1 leader_uuid: "6e6590a0bb9f462faf563903a1db0bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6590a0bb9f462faf563903a1db0bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } } }
I20260620 14:20:40.495637 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:40.495591 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:40.496141 4021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:40.498694 4021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:40.503728 4021 catalog_manager.cc:1347] Generated new cluster ID: 606878cdcf134c8096a30b0ff891b099
I20260620 14:20:40.503816 4021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:40.517514 4021 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:40.518800 4021 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:40.528860 4021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6590a0bb9f462faf563903a1db0bf1: Generated new TSK 0
I20260620 14:20:40.529692 4021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:20:40.562722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:40.677989 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:40.678252 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:40.678333 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:40.682425 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:40.682503 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:40.682615 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:40.687947 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:40.689021 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:40.690090 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:40.699795 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:40.699802 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:40.699813 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:40.699887 4033 server_base.cc:1034] running on GCE node
I20260620 14:20:40.700418 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:40.701085 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:40.702306 4033 hybrid_clock.cc:648] HybridClock initialized: now 1781965240702275 us; error 50 us; skew 500 ppm
Jun 20 14:20:40 dist-test-slave-87nc krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:40.705086 4033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:40.705956 4033 webserver.cc:460] Webserver started at http://127.0.98.1:35523/ using document root <none> and password file <none>
I20260620 14:20:40.706521 4033 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:40.706614 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:40.706856 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:40.708952 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "48b99da295184b26ac2a0f147594de7d"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-87nc"
server_key: "4e58978e84343b5f593fab3f61a6de34"
server_key_iv: "c5330e11f6b35742229f265bf208b2a0"
server_key_version: "encryptionkey@0"
I20260620 14:20:40.709517 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48b99da295184b26ac2a0f147594de7d"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-87nc"
server_key: "4e58978e84343b5f593fab3f61a6de34"
server_key_iv: "c5330e11f6b35742229f265bf208b2a0"
server_key_version: "encryptionkey@0"
I20260620 14:20:40.713353 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:20:40.715749 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:40.717008 4033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:40.717130 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "48b99da295184b26ac2a0f147594de7d"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-87nc"
server_key: "4e58978e84343b5f593fab3f61a6de34"
server_key_iv: "c5330e11f6b35742229f265bf208b2a0"
server_key_version: "encryptionkey@0"
I20260620 14:20:40.717234 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:40.736351 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:40.739115 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:40.739315 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:40.739966 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:40.740790 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:40.740835 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:40.740901 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:40.740931 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:40.750106 4033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35921
I20260620 14:20:40.750128 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35921 every 8 connection(s)
I20260620 14:20:40.750995 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:40.759117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 4033
I20260620 14:20:40.759255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:40.759605 392 external_mini_cluster.cc:1428] Setting key 6472bda4ae1e1175731581154b8cf41e
Jun 20 14:20:40 dist-test-slave-87nc krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:40.765103 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:40.752815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38291 (local address 127.0.98.62:37391)
0620 14:20:40.752966 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:40.752977 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:40.753884 (+ 907us) server_negotiation.cc:407] Connection header received
0620 14:20:40.755178 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:40.755181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:40.755224 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:40.755295 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:40.756434 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.756953 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.758069 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.758280 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.758353 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:40.761064 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:40.761068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:40.761071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:40.761098 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:40.762886 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:40.763386 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:40.763389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:40.763391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:40.763492 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:40.763863 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:40.763870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:40.763874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:40.764164 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:40.764276 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:40.764781 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 14:20:40.764904 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:20:40.765923 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:40.753272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37391 (local address 127.0.98.1:38291)
0620 14:20:40.753704 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:40.753745 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:40.754985 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:40.755425 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:40.755431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:40.755828 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:40.756308 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:40.756321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.757086 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.757089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:40.757951 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:40.757959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.758403 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.758406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:40.758545 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:40.759136 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:40.759158 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:40.760893 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:40.763004 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:40.763009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:40.763024 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:40.763282 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:40.763586 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:40.763588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:40.763589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:40.763720 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:40.764274 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:40.764281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:40.764539 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 14:20:40.764729 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 14:20:40.765077 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260620 14:20:40.767318 4162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37391
I20260620 14:20:40.767570 4162 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:40.768110 4162 heartbeater.cc:511] Master 127.0.98.62:37391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:20:40.769795 3971 ts_manager.cc:194] Registered new tserver with Master: 48b99da295184b26ac2a0f147594de7d (127.0.98.1:35921)
I20260620 14:20:40.770972 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:20:40.786675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:40.897560 4167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:40.897825 4167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:40.897868 4167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:40.901966 4167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:40.902026 4167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:40.902086 4167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:40.907397 4167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:40.908406 4167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:40.909345 4167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:40.918797 4172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:40.918769 4173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:40.918946 4167 server_base.cc:1034] running on GCE node
W20260620 14:20:40.918792 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:40.919486 4167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:40.920171 4167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:40.921375 4167 hybrid_clock.cc:648] HybridClock initialized: now 1781965240921361 us; error 33 us; skew 500 ppm
Jun 20 14:20:40 dist-test-slave-87nc krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:40.924074 4167 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:40.924932 4167 webserver.cc:460] Webserver started at http://127.0.98.2:40851/ using document root <none> and password file <none>
I20260620 14:20:40.925490 4167 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:40.925576 4167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:40.925804 4167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:40.927755 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "68f03e5f5b554d22a9bbea72b49b459c"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-87nc"
server_key: "03bfa52db26ff248095ee2cf5c93e870"
server_key_iv: "73a6eb3f3b55f4c834f6e2945bed8081"
server_key_version: "encryptionkey@0"
I20260620 14:20:40.928400 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "68f03e5f5b554d22a9bbea72b49b459c"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-87nc"
server_key: "03bfa52db26ff248095ee2cf5c93e870"
server_key_iv: "73a6eb3f3b55f4c834f6e2945bed8081"
server_key_version: "encryptionkey@0"
I20260620 14:20:40.932220 4167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:20:40.934685 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:40.935948 4167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:40.936069 4167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "68f03e5f5b554d22a9bbea72b49b459c"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-87nc"
server_key: "03bfa52db26ff248095ee2cf5c93e870"
server_key_iv: "73a6eb3f3b55f4c834f6e2945bed8081"
server_key_version: "encryptionkey@0"
I20260620 14:20:40.936173 4167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:40.957293 4167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:40.960570 4167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:40.960727 4167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:40.961251 4167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:40.962023 4167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:40.962069 4167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:40.962108 4167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:40.962129 4167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:40.970419 4167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42479
I20260620 14:20:40.970440 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42479 every 8 connection(s)
I20260620 14:20:40.971333 4167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:40.972954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 4167
I20260620 14:20:40.973081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:40.973390 392 external_mini_cluster.cc:1428] Setting key 29958f079845d8622374c8e576b9c25a
Jun 20 14:20:40 dist-test-slave-87nc krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:20:40.987577 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:40.973249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48461 (local address 127.0.98.62:37391)
0620 14:20:40.973406 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:40.973411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:40.974389 (+ 978us) server_negotiation.cc:407] Connection header received
0620 14:20:40.976407 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:40.976409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:40.976451 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:40.976514 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:40.977823 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.978342 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.980249 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.980489 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.980579 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:40.983636 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:40.983639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:40.983641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:40.983664 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:40.985306 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:40.985868 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:40.985870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:40.985872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:40.985966 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:40.986303 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:40.986309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:40.986314 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:40.986586 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:40.986692 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:40.987194 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 14:20:40.987339 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:20:40.988556 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:40.973696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37391 (local address 127.0.98.2:48461)
0620 14:20:40.974247 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:40.974290 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:40.976156 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:40.976738 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:40.976751 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:40.977140 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:40.977623 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:40.977636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.978480 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.978486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:40.980091 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:40.980123 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.980666 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.980672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:40.980934 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:40.981584 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:40.981607 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:40.983446 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:40.985403 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:40.985408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:40.985420 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:40.985773 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:40.986071 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:40.986073 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:40.986075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:40.986208 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:40.986687 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:40.986692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:40.986929 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 14:20:40.987092 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 14:20:40.987497 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":146,"threads_started":1}
I20260620 14:20:40.989833 4296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37391
I20260620 14:20:40.990156 4296 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:40.990783 4296 heartbeater.cc:511] Master 127.0.98.62:37391 requested a full tablet report, sending...
I20260620 14:20:40.992002 3971 ts_manager.cc:194] Registered new tserver with Master: 68f03e5f5b554d22a9bbea72b49b459c (127.0.98.2:42479)
I20260620 14:20:40.992578 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:20:41.002337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:41.111311 4301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:41.111658 4301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:41.111734 4301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:41.116082 4301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:41.116163 4301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:41.116248 4301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:41.121673 4301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:41.122764 4301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:41.123874 4301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:41.133582 4307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:41.133846 4301 server_base.cc:1034] running on GCE node
W20260620 14:20:41.133582 4309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:41.133598 4306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:41.134400 4301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:41.135097 4301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:41.136281 4301 hybrid_clock.cc:648] HybridClock initialized: now 1781965241136261 us; error 36 us; skew 500 ppm
Jun 20 14:20:41 dist-test-slave-87nc krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:41.138926 4301 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:41.140021 4301 webserver.cc:460] Webserver started at http://127.0.98.3:44357/ using document root <none> and password file <none>
I20260620 14:20:41.140623 4301 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:41.140717 4301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:41.140949 4301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:41.142988 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "c7ada829147749c88773defe3bca79da"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-87nc"
server_key: "ea8bd9536de73557c28a17d0c35ef6f1"
server_key_iv: "165ecbdadb09cb97baf369a0a49cb8b4"
server_key_version: "encryptionkey@0"
I20260620 14:20:41.143564 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c7ada829147749c88773defe3bca79da"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-87nc"
server_key: "ea8bd9536de73557c28a17d0c35ef6f1"
server_key_iv: "165ecbdadb09cb97baf369a0a49cb8b4"
server_key_version: "encryptionkey@0"
I20260620 14:20:41.147440 4301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:20:41.149865 4316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.151069 4301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:41.151211 4301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "c7ada829147749c88773defe3bca79da"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-87nc"
server_key: "ea8bd9536de73557c28a17d0c35ef6f1"
server_key_iv: "165ecbdadb09cb97baf369a0a49cb8b4"
server_key_version: "encryptionkey@0"
I20260620 14:20:41.151327 4301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:41.171118 4301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:41.174036 4301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:41.174242 4301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:41.174835 4301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:41.175649 4301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:41.175722 4301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.175815 4301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:41.175870 4301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.184888 4301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39173
I20260620 14:20:41.184909 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39173 every 8 connection(s)
I20260620 14:20:41.185794 4301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:41.188555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 4301
I20260620 14:20:41.188681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:41.189003 392 external_mini_cluster.cc:1428] Setting key c0a1f37947cd1f7de8a03dfae974dcdb
Jun 20 14:20:41 dist-test-slave-87nc krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:41.199621 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.187577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55227 (local address 127.0.98.62:37391)
0620 14:20:41.187731 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:41.187735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:41.188742 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 14:20:41.190461 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:41.190464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:41.190518 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:41.190600 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:41.191881 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.192404 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.193523 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.193742 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.193873 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.196473 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:41.196477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:41.196479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:41.196502 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:41.197863 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.198363 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.198366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.198368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.198443 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.198736 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.198738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.198740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.198895 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:41.198978 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:41.199413 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 14:20:41.199508 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:20:41.200440 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.188053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37391 (local address 127.0.98.3:55227)
0620 14:20:41.188573 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:41.188633 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:41.190251 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:41.190779 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:41.190786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:41.191192 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:41.191682 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:41.191694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.192526 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.192530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:41.193390 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:41.193398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.193885 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.193889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:41.194114 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.194783 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:41.194799 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:41.196333 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:41.197978 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:41.197982 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:41.197993 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:41.198242 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:41.198524 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:41.198526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:41.198528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:41.198652 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:41.198996 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:41.199001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:41.199217 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 14:20:41.199357 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 14:20:41.199598 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260620 14:20:41.201545 4430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37391
I20260620 14:20:41.201797 4430 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:41.202327 4430 heartbeater.cc:511] Master 127.0.98.62:37391 requested a full tablet report, sending...
I20260620 14:20:41.203336 3971 ts_manager.cc:194] Registered new tserver with Master: c7ada829147749c88773defe3bca79da (127.0.98.3:39173)
I20260620 14:20:41.203922 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55227
I20260620 14:20:41.214257 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:20:41 dist-test-slave-87nc krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:20:41 dist-test-slave-87nc krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:41.238761 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.229843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.0.98.1:35921)
0620 14:20:41.230120 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:41.230133 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:41.230155 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:20:41.230203 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:41.230207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:41.230387 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:41.230504 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:41.231117 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.232017 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.232805 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.233167 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.233309 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.235746 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:41.235754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:41.235809 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:41.235836 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:41.237264 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.237631 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.237635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.237636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.237706 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.237918 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.237921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.237922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.238148 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:41.238307 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:41.238433 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 14:20:41.238546 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
Jun 20 14:20:41 dist-test-slave-87nc krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:41.249408 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.241641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.0.98.62:37391)
0620 14:20:41.241773 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:41.241777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:41.241790 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:20:41.241872 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:41.241874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:41.241911 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:41.241982 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:41.242498 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.242976 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.243679 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.243966 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.244046 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.246697 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:41.246700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:41.246702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:41.246723 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:41.247989 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.248375 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.248377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.248378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.248443 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.248649 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.248651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.248652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.248851 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:41.248959 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:41.249101 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:20:41.249197 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 14:20:41.249878 3971 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58776
I20260620 14:20:41.250398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 4033
I20260620 14:20:41.256879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 4167
I20260620 14:20:41.262225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 4301
I20260620 14:20:41.267285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 3939
2026-06-20T14:20:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2115 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:41 dist-test-slave-87nc krb5kdc[4442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:41 dist-test-slave-87nc krb5kdc[4442](info): set up 2 sockets
Jun 20 14:20:41 dist-test-slave-87nc krb5kdc[4442](info): commencing operation
krb5kdc: starting...
W20260620 14:20:42.308148 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:42 dist-test-slave-87nc krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:42.426461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34963
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:42.534624 4455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:42.534874 4455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:42.534915 4455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:42.538877 4455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:42.538930 4455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:42.538955 4455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:42.538972 4455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:42.538987 4455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:42.544224 4455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:42.545243 4455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:42.546262 4455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:42.554260 4463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:42.554279 4460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:42.554260 4461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:42.554486 4455 server_base.cc:1034] running on GCE node
I20260620 14:20:42.554999 4455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:42.556107 4455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:42.557266 4455 hybrid_clock.cc:648] HybridClock initialized: now 1781965242557255 us; error 27 us; skew 500 ppm
Jun 20 14:20:42 dist-test-slave-87nc krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:42.563443 4455 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:42.564244 4455 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:42.564312 4455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:42.564486 4455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:42.566514 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae"
format_stamp: "Formatted at 2026-06-20 14:20:42 on dist-test-slave-87nc"
server_key: "faf952c5cc3a535e8863b8fe936a0f8c"
server_key_iv: "b0de79a64febd5679c0c7abad27e9736"
server_key_version: "encryptionkey@0"
I20260620 14:20:42.567035 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae"
format_stamp: "Formatted at 2026-06-20 14:20:42 on dist-test-slave-87nc"
server_key: "faf952c5cc3a535e8863b8fe936a0f8c"
server_key_iv: "b0de79a64febd5679c0c7abad27e9736"
server_key_version: "encryptionkey@0"
I20260620 14:20:42.570638 4455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:20:42.572973 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:42.574271 4455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:42.574363 4455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae"
format_stamp: "Formatted at 2026-06-20 14:20:42 on dist-test-slave-87nc"
server_key: "faf952c5cc3a535e8863b8fe936a0f8c"
server_key_iv: "b0de79a64febd5679c0c7abad27e9736"
server_key_version: "encryptionkey@0"
I20260620 14:20:42.574455 4455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:42.592259 4455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:42.595029 4455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:42.595180 4455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:42.601624 4455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34963
I20260620 14:20:42.601890 4521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34963 every 8 connection(s)
I20260620 14:20:42.602262 4455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:42.602566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 4455
I20260620 14:20:42.602694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:42.602983 392 external_mini_cluster.cc:1428] Setting key d0d378efe6107974a24992d4b94025a6
I20260620 14:20:42.605206 4522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:42.611598 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae: Bootstrap starting.
Jun 20 14:20:42 dist-test-slave-87nc krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:42.613932 4522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:42.614621 4522 log.cc:824] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:42.616541 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae: No bootstrap required, opened a new log
I20260620 14:20:42.617096 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.604333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.0.98.62:34963)
0620 14:20:42.604727 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:42.604751 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:42.604787 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:20:42.605479 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:42.605498 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:42.605814 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:42.606136 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:42.607030 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.608081 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.609227 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.609683 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.609795 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.612592 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:42.612597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:42.612610 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:42.612637 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:42.614292 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:42.614728 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:42.614736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:42.614743 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:42.614871 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:42.615167 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:42.615172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:42.615174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:42.615557 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:42.615756 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:42.616082 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:20:42.616351 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":140,"threads_started":1}
I20260620 14:20:42.620046 4522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } }
I20260620 14:20:42.620271 4522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:42.620353 4522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c1e5b03494446fa0cdb6ebe0ada5ae, State: Initialized, Role: FOLLOWER
I20260620 14:20:42.620795 4522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } }
I20260620 14:20:42.620944 4522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:42.621009 4522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:42.621106 4522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:42.621949 4522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } }
I20260620 14:20:42.622270 4522 leader_election.cc:302] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c1e5b03494446fa0cdb6ebe0ada5ae; no voters:
I20260620 14:20:42.622572 4522 leader_election.cc:288] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:42.622666 4527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:42.622892 4527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [term 1 LEADER]: Becoming Leader. State: Replica: 33c1e5b03494446fa0cdb6ebe0ada5ae, State: Running, Role: LEADER
I20260620 14:20:42.623260 4527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } }
I20260620 14:20:42.623817 4522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:42.624890 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c1e5b03494446fa0cdb6ebe0ada5ae. Latest consensus state: current_term: 1 leader_uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } } }
I20260620 14:20:42.625028 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:42.625576 4536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:42.626214 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c1e5b03494446fa0cdb6ebe0ada5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } } }
I20260620 14:20:42.626822 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:42.629616 4536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:42.636068 4536 catalog_manager.cc:1347] Generated new cluster ID: f951816fc5a34b209280a77a78402a7b
I20260620 14:20:42.636145 4536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:42.644695 4536 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:42.645565 4536 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:42.653954 4536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae: Generated new TSK 0
I20260620 14:20:42.654667 4536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:20:42.686309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34963
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:42.797063 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:42.797374 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:42.797452 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:42.801476 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:42.801563 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:42.801672 4548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:42.807085 4548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:42.808197 4548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:42.809253 4548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:42.818696 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:42.818648 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:42.818650 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:42.818982 4548 server_base.cc:1034] running on GCE node
I20260620 14:20:42.819419 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:42.820150 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:42.821310 4548 hybrid_clock.cc:648] HybridClock initialized: now 1781965242821291 us; error 32 us; skew 500 ppm
Jun 20 14:20:42 dist-test-slave-87nc krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:42.824203 4548 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:42.825023 4548 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:42.825119 4548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:42.825352 4548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:42.827431 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "1ffce03f48b342eba8f55d713a3d9357"
format_stamp: "Formatted at 2026-06-20 14:20:42 on dist-test-slave-87nc"
server_key: "ca8857b6733cc28d287349c4b5b1b64a"
server_key_iv: "7afd5b4546217958306f445558ab56fe"
server_key_version: "encryptionkey@0"
I20260620 14:20:42.828042 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1ffce03f48b342eba8f55d713a3d9357"
format_stamp: "Formatted at 2026-06-20 14:20:42 on dist-test-slave-87nc"
server_key: "ca8857b6733cc28d287349c4b5b1b64a"
server_key_iv: "7afd5b4546217958306f445558ab56fe"
server_key_version: "encryptionkey@0"
I20260620 14:20:42.831622 4548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:20:42.834080 4562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:42.835271 4548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:42.835398 4548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "1ffce03f48b342eba8f55d713a3d9357"
format_stamp: "Formatted at 2026-06-20 14:20:42 on dist-test-slave-87nc"
server_key: "ca8857b6733cc28d287349c4b5b1b64a"
server_key_iv: "7afd5b4546217958306f445558ab56fe"
server_key_version: "encryptionkey@0"
I20260620 14:20:42.835505 4548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:42.850844 4548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:42.853721 4548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:42.853921 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:42.854489 4548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:42.855293 4548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:42.855363 4548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:42.855430 4548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:42.855479 4548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:42.864223 4548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43943
I20260620 14:20:42.864240 4675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43943 every 8 connection(s)
I20260620 14:20:42.864804 4548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:42.873018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 4548
I20260620 14:20:42.873152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:42.873410 392 external_mini_cluster.cc:1428] Setting key e0a27d9c5916e8a7025963ee9f9b9c60
Jun 20 14:20:42 dist-test-slave-87nc krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:42.877951 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.866503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45155 (local address 127.0.98.62:34963)
0620 14:20:42.866642 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:42.866645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:42.867438 (+ 793us) server_negotiation.cc:407] Connection header received
0620 14:20:42.868755 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:42.868758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:42.868803 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:42.868876 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:42.869951 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.870475 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.871561 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.871865 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.871967 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.874352 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:42.874355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:42.874357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:42.874380 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:42.875744 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:42.876406 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:42.876411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:42.876414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:42.876526 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:42.876833 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:42.876838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:42.876840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:42.877094 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:42.877201 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:42.877682 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 14:20:42.877833 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:20:42.879027 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.866864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34963 (local address 127.0.98.1:45155)
0620 14:20:42.867286 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:42.867326 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:42.868572 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:42.869029 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:42.869035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:42.869373 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:42.869831 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.869843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.870572 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.870575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.871438 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.871446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.871992 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.871995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.872129 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.872722 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:42.872738 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:42.874190 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:42.875918 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:42.875926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:42.875942 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:42.876298 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:42.876622 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:42.876624 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:42.876625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:42.876748 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:42.877194 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:42.877199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:42.877415 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 14:20:42.877593 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 14:20:42.877952 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260620 14:20:42.880401 4676 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34963
I20260620 14:20:42.880635 4676 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:42.881163 4676 heartbeater.cc:511] Master 127.0.98.62:34963 requested a full tablet report, sending...
I20260620 14:20:42.882856 4486 ts_manager.cc:194] Registered new tserver with Master: 1ffce03f48b342eba8f55d713a3d9357 (127.0.98.1:43943)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:20:42.884296 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:20:42.903254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34963
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:43.013994 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:43.014268 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:43.014315 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:43.018472 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:43.018536 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:43.018625 4681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:43.023733 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:43.024914 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:43.026027 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:43.035423 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:43.035436 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:43.035492 4681 server_base.cc:1034] running on GCE node
W20260620 14:20:43.035445 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:43.036136 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:43.036698 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:43.037854 4681 hybrid_clock.cc:648] HybridClock initialized: now 1781965243037836 us; error 28 us; skew 500 ppm
Jun 20 14:20:43 dist-test-slave-87nc krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:43.040733 4681 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:43.041463 4681 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:43.041527 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:43.041734 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:43.043713 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "76ca3576ce0b4b85867b16af0131d7b1"
format_stamp: "Formatted at 2026-06-20 14:20:43 on dist-test-slave-87nc"
server_key: "e27fd50f8bfc42fc72ea768c01eb936d"
server_key_iv: "a9cc113c4e1b3e37d92c96044c709685"
server_key_version: "encryptionkey@0"
I20260620 14:20:43.044397 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "76ca3576ce0b4b85867b16af0131d7b1"
format_stamp: "Formatted at 2026-06-20 14:20:43 on dist-test-slave-87nc"
server_key: "e27fd50f8bfc42fc72ea768c01eb936d"
server_key_iv: "a9cc113c4e1b3e37d92c96044c709685"
server_key_version: "encryptionkey@0"
I20260620 14:20:43.048002 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:20:43.050282 4695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:43.051491 4681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:43.051622 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "76ca3576ce0b4b85867b16af0131d7b1"
format_stamp: "Formatted at 2026-06-20 14:20:43 on dist-test-slave-87nc"
server_key: "e27fd50f8bfc42fc72ea768c01eb936d"
server_key_iv: "a9cc113c4e1b3e37d92c96044c709685"
server_key_version: "encryptionkey@0"
I20260620 14:20:43.051733 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:43.069665 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:43.072721 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:43.072927 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:43.073529 4681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:43.074374 4681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:43.074457 4681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:43.074549 4681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:43.074600 4681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:43.083724 4681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36439
I20260620 14:20:43.083741 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36439 every 8 connection(s)
I20260620 14:20:43.084491 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:43.089169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 4681
I20260620 14:20:43.089298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:43.089613 392 external_mini_cluster.cc:1428] Setting key c855ff25a1d668d658c05ca62bc1b947
Jun 20 14:20:43 dist-test-slave-87nc krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:43.099247 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.086185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44217 (local address 127.0.98.62:34963)
0620 14:20:43.086312 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:43.086316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:43.087149 (+ 833us) server_negotiation.cc:407] Connection header received
0620 14:20:43.088519 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:43.088521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:43.088564 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:43.088627 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:43.090088 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.090783 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.092619 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.092856 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.092970 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.095336 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:43.095343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:43.095350 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:43.095383 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:43.097176 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:43.097685 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:43.097692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:43.097696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:43.097804 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:43.098124 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:43.098129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:43.098133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:43.098372 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:43.098485 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:43.098939 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 14:20:43.099083 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:20:43.100030 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.086592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34963 (local address 127.0.98.2:44217)
0620 14:20:43.087004 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:43.087044 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:43.088311 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:43.088766 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:43.088772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:43.089252 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:43.089927 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.089947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.090937 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.090940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.092500 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.092514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.093008 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.093012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.093173 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.093766 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:43.093782 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:43.095169 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:43.097318 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:43.097323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:43.097334 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:43.097584 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:43.097906 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:43.097909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:43.097910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:43.098029 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:43.098474 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:43.098479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:43.098694 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 14:20:43.098842 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 14:20:43.099080 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":99,"threads_started":1}
I20260620 14:20:43.101266 4809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34963
I20260620 14:20:43.101521 4809 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:20:43.102026 4809 heartbeater.cc:511] Master 127.0.98.62:34963 requested a full tablet report, sending...
I20260620 14:20:43.103039 4486 ts_manager.cc:194] Registered new tserver with Master: 76ca3576ce0b4b85867b16af0131d7b1 (127.0.98.2:36439)
I20260620 14:20:43.103597 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:20:43.116123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34963
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:43.228111 4814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:43.228387 4814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:43.228482 4814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:43.232580 4814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:43.232646 4814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:43.232751 4814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:43.238078 4814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:43.239264 4814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:43.240433 4814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:43.249871 4822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:43.249854 4819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:43.249857 4820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:43.249963 4814 server_base.cc:1034] running on GCE node
I20260620 14:20:43.250438 4814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:43.251103 4814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:43.252290 4814 hybrid_clock.cc:648] HybridClock initialized: now 1781965243252264 us; error 34 us; skew 500 ppm
Jun 20 14:20:43 dist-test-slave-87nc krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:43.255064 4814 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:43.255937 4814 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:43.256032 4814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:43.256240 4814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:43.258237 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "37d20c04811e464fba5134eb5c296405"
format_stamp: "Formatted at 2026-06-20 14:20:43 on dist-test-slave-87nc"
server_key: "52182f5f307b25ea95d7dfecdd2b2d6c"
server_key_iv: "211371cf6dc00ebd7d7714f80912e857"
server_key_version: "encryptionkey@0"
I20260620 14:20:43.258806 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "37d20c04811e464fba5134eb5c296405"
format_stamp: "Formatted at 2026-06-20 14:20:43 on dist-test-slave-87nc"
server_key: "52182f5f307b25ea95d7dfecdd2b2d6c"
server_key_iv: "211371cf6dc00ebd7d7714f80912e857"
server_key_version: "encryptionkey@0"
I20260620 14:20:43.262459 4814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:20:43.264837 4828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:43.266120 4814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:43.266280 4814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "37d20c04811e464fba5134eb5c296405"
format_stamp: "Formatted at 2026-06-20 14:20:43 on dist-test-slave-87nc"
server_key: "52182f5f307b25ea95d7dfecdd2b2d6c"
server_key_iv: "211371cf6dc00ebd7d7714f80912e857"
server_key_version: "encryptionkey@0"
I20260620 14:20:43.266395 4814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:43.300875 4814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:43.303859 4814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:43.304067 4814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:43.304669 4814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:43.305536 4814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:43.305610 4814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:43.305682 4814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:43.305734 4814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:43.314239 4814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42967
I20260620 14:20:43.314260 4941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42967 every 8 connection(s)
I20260620 14:20:43.314823 4814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:43.322664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 4814
I20260620 14:20:43.322788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:43.323093 392 external_mini_cluster.cc:1428] Setting key 783205751a510fc0bffdf5c6f7010746
Jun 20 14:20:43 dist-test-slave-87nc krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:43.328788 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.316568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36881 (local address 127.0.98.62:34963)
0620 14:20:43.316746 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:43.316752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:43.317821 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 14:20:43.319233 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:43.319236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:43.319285 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:43.319361 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:43.320476 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.320986 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.322057 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.322264 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.322338 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.325456 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:43.325459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:43.325462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:43.325485 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:43.326932 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:43.327382 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:43.327384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:43.327385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:43.327464 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:43.327746 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:43.327748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:43.327749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:43.327971 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:43.328047 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:43.328576 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 14:20:43.328673 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:20:43.329711 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.317075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34963 (local address 127.0.98.3:36881)
0620 14:20:43.317658 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:43.317716 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:43.319027 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:43.319492 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:43.319499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:43.319871 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:43.320340 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.320352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.321091 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.321093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.321955 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.321963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.322385 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.322388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.322531 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.323211 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:43.323232 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:43.325266 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:43.327030 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:43.327034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:43.327047 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:43.327298 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:43.327541 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:43.327543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:43.327544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:43.327672 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:43.328056 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:43.328062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:43.328357 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 14:20:43.328531 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 14:20:43.328783 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":149,"threads_started":1}
I20260620 14:20:43.330809 4942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34963
I20260620 14:20:43.331030 4942 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:43.331427 4942 heartbeater.cc:511] Master 127.0.98.62:34963 requested a full tablet report, sending...
I20260620 14:20:43.332437 4486 ts_manager.cc:194] Registered new tserver with Master: 37d20c04811e464fba5134eb5c296405 (127.0.98.3:42967)
I20260620 14:20:43.332952 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36881
I20260620 14:20:43.336807 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:20:43.345490 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.338577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.0.98.62:34963)
0620 14:20:43.338720 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:43.338726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:43.338871 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:20:43.339011 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:43.339013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:43.339048 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:43.339144 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:43.339745 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.340304 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.341069 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.341266 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.341336 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.342516 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:43.342519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:43.342521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:43.342541 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:43.344045 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:43.344396 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:43.344399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:43.344400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:43.344474 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:43.344702 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:43.344704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:43.344705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:43.344890 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:43.344988 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:43.345181 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 14:20:43.345276 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:20:43.348088 4486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:43.350924 4486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:43.364880 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.358774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42967 (local address 127.0.0.1:47242)
0620 14:20:43.359385 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:43.359435 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:43.359565 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:43.360719 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:43.360725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:43.360747 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:43.360931 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.360943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.362478 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.362482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.363464 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.363473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.364441 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.364443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.364517 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.364547 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:20:43.364639 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:20:43.364723 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":35,"spinlock_wait_cycles":10112,"thread_start_us":78,"threads_started":1}
I20260620 14:20:43.364938 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.359894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.0.98.3:42967)
0620 14:20:43.360225 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:43.360240 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:43.360262 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:20:43.360340 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:43.360346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:43.360556 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:43.360728 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:43.361115 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.362329 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.363578 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.364328 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.364439 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.364586 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:43.364674 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:20:43.364769 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":113,"threads_started":1}
I20260620 14:20:43.365306 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.359489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36439 (local address 127.0.0.1:41762)
0620 14:20:43.359766 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:43.359777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:43.359879 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:43.361004 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:43.361007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:43.361022 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:43.361197 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.361203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.362597 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.362600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.363854 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.363861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.364877 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.364880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.364985 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.365008 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:20:43.365086 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:20:43.365171 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":84,"spinlock_wait_cycles":7040,"thread_start_us":94,"threads_started":1}
I20260620 14:20:43.365782 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.360273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.0.98.2:36439)
0620 14:20:43.360545 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:43.360563 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:43.360581 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:20:43.360640 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:43.360664 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:43.360858 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:43.361018 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:43.361330 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.362478 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.363976 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.364757 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.364868 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.365005 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:43.365552 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 14:20:43.365645 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20260620 14:20:43.366770 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.358886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43943 (local address 127.0.0.1:50050)
0620 14:20:43.359738 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:43.359750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:43.359879 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:43.360749 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:43.360752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:43.360774 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:43.361137 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.361145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.362123 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.362129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.363838 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.363848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.366396 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.366400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.366486 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.366506 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:20:43.366570 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:20:43.366638 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":223,"spinlock_wait_cycles":3200,"thread_start_us":104,"threads_started":1}
I20260620 14:20:43.366822 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.359220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.0.98.1:43943)
0620 14:20:43.360359 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:43.360365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:43.360383 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:20:43.360455 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:43.360460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:43.360601 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:43.360919 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:43.361261 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.361999 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.365072 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.366264 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.366400 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.366544 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:43.366597 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:20:43.366679 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1043,"thread_start_us":92,"threads_started":1}
I20260620 14:20:43.368301 4743 tablet_service.cc:1467] Processing CreateTablet for tablet 544a5e3bdc2147c0b30a0ffafa253aa6 (DEFAULT_TABLE table=test-table [id=0dd143f7209f4594a79f8be273cc9d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:43.368301 4876 tablet_service.cc:1467] Processing CreateTablet for tablet 544a5e3bdc2147c0b30a0ffafa253aa6 (DEFAULT_TABLE table=test-table [id=0dd143f7209f4594a79f8be273cc9d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:43.369459 4876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544a5e3bdc2147c0b30a0ffafa253aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:43.369459 4743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544a5e3bdc2147c0b30a0ffafa253aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:43.370394 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 544a5e3bdc2147c0b30a0ffafa253aa6 (DEFAULT_TABLE table=test-table [id=0dd143f7209f4594a79f8be273cc9d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:43.371349 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544a5e3bdc2147c0b30a0ffafa253aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:43.377487 4960 tablet_bootstrap.cc:492] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: Bootstrap starting.
I20260620 14:20:43.377614 4959 tablet_bootstrap.cc:492] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: Bootstrap starting.
I20260620 14:20:43.378023 4961 tablet_bootstrap.cc:492] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357: Bootstrap starting.
I20260620 14:20:43.379729 4959 tablet_bootstrap.cc:654] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:43.379946 4961 tablet_bootstrap.cc:654] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:43.380795 4961 log.cc:824] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:43.380839 4959 log.cc:824] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:43.381717 4960 tablet_bootstrap.cc:654] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:43.382861 4959 tablet_bootstrap.cc:492] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: No bootstrap required, opened a new log
I20260620 14:20:43.382861 4961 tablet_bootstrap.cc:492] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357: No bootstrap required, opened a new log
I20260620 14:20:43.382905 4960 log.cc:824] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:43.383064 4961 ts_tablet_manager.cc:1397] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:20:43.383064 4959 ts_tablet_manager.cc:1397] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 14:20:43.384641 4960 tablet_bootstrap.cc:492] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: No bootstrap required, opened a new log
I20260620 14:20:43.384843 4960 ts_tablet_manager.cc:1397] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 14:20:43.387125 4959 raft_consensus.cc:348] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } }
I20260620 14:20:43.387431 4959 raft_consensus.cc:374] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:43.387300 4961 raft_consensus.cc:348] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } }
I20260620 14:20:43.387519 4959 raft_consensus.cc:729] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ca3576ce0b4b85867b16af0131d7b1, State: Initialized, Role: FOLLOWER
I20260620 14:20:43.387557 4961 raft_consensus.cc:374] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:43.387642 4961 raft_consensus.cc:729] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffce03f48b342eba8f55d713a3d9357, State: Initialized, Role: FOLLOWER
I20260620 14:20:43.387588 4960 raft_consensus.cc:348] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } }
I20260620 14:20:43.387902 4960 raft_consensus.cc:374] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:43.387975 4960 raft_consensus.cc:729] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d20c04811e464fba5134eb5c296405, State: Initialized, Role: FOLLOWER
I20260620 14:20:43.388094 4959 consensus_queue.cc:260] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } }
I20260620 14:20:43.388094 4961 consensus_queue.cc:260] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } }
I20260620 14:20:43.388425 4960 consensus_queue.cc:260] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } }
I20260620 14:20:43.389132 4961 ts_tablet_manager.cc:1428] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:20:43.389132 4959 ts_tablet_manager.cc:1428] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:20:43.389348 4960 ts_tablet_manager.cc:1428] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 14:20:43.389693 4942 heartbeater.cc:503] Master 127.0.98.62:34963 was elected leader, sending a full tablet report...
I20260620 14:20:43.389808 4676 heartbeater.cc:503] Master 127.0.98.62:34963 was elected leader, sending a full tablet report...
I20260620 14:20:43.390910 4809 heartbeater.cc:503] Master 127.0.98.62:34963 was elected leader, sending a full tablet report...
W20260620 14:20:43.565827 4943 tablet.cc:2369] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:43.585561 4810 tablet.cc:2369] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:43.616142 4677 tablet.cc:2369] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:43.768538 4967 raft_consensus.cc:482] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:43.768749 4967 raft_consensus.cc:504] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } }
I20260620 14:20:43.769876 4967 leader_election.cc:288] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ffce03f48b342eba8f55d713a3d9357 (127.0.98.1:43943), 76ca3576ce0b4b85867b16af0131d7b1 (127.0.98.2:36439)
I20260620 14:20:43.773609 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.770107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43943 (local address 127.0.98.3:52543)
0620 14:20:43.770338 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:43.770356 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:43.770470 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:43.770748 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:43.770751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:43.770766 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:43.770884 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.770889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.771653 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.771657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.772510 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.772519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.773164 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.773168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.773271 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.773296 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:20:43.773383 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:20:43.773479 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 14:20:43.773609 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.770270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52543 (local address 127.0.98.1:43943)
0620 14:20:43.770403 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:43.770408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:43.770426 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:20:43.770597 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:43.770600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:43.770644 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:43.770736 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:43.771000 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.771537 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.772642 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.773041 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.773159 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.773220 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:43.773409 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 14:20:43.773480 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:20:43.773819 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.770155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36439 (local address 127.0.98.3:42103)
0620 14:20:43.770542 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:43.770561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:43.770674 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:43.770947 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:43.770949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:43.770964 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:43.771084 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.771089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.771911 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.771916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.772731 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:43.772739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.773398 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.773409 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:43.773512 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.773534 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:20:43.773620 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:20:43.773683 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":98,"threads_started":1}
I20260620 14:20:43.773808 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.770323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42103 (local address 127.0.98.2:36439)
0620 14:20:43.770495 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:43.770502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:43.770631 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:20:43.770785 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:43.770791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:43.770836 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:43.770936 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:43.771190 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.771805 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.772848 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.773239 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.773369 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.773428 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:43.773635 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 14:20:43.773697 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 14:20:43.774418 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544a5e3bdc2147c0b30a0ffafa253aa6" candidate_uuid: "37d20c04811e464fba5134eb5c296405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffce03f48b342eba8f55d713a3d9357" is_pre_election: true
I20260620 14:20:43.774423 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544a5e3bdc2147c0b30a0ffafa253aa6" candidate_uuid: "37d20c04811e464fba5134eb5c296405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ca3576ce0b4b85867b16af0131d7b1" is_pre_election: true
I20260620 14:20:43.774734 4763 raft_consensus.cc:2454] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37d20c04811e464fba5134eb5c296405 in term 0.
I20260620 14:20:43.774734 4630 raft_consensus.cc:2454] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37d20c04811e464fba5134eb5c296405 in term 0.
I20260620 14:20:43.775250 4830 leader_election.cc:302] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ffce03f48b342eba8f55d713a3d9357, 37d20c04811e464fba5134eb5c296405, 76ca3576ce0b4b85867b16af0131d7b1; no voters:
I20260620 14:20:43.775544 4967 raft_consensus.cc:2781] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:43.775642 4967 raft_consensus.cc:482] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:43.775724 4967 raft_consensus.cc:3037] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:43.776800 4967 raft_consensus.cc:504] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } }
I20260620 14:20:43.777149 4967 leader_election.cc:288] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [CANDIDATE]: Term 1 election: Requested vote from peers 1ffce03f48b342eba8f55d713a3d9357 (127.0.98.1:43943), 76ca3576ce0b4b85867b16af0131d7b1 (127.0.98.2:36439)
I20260620 14:20:43.777540 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544a5e3bdc2147c0b30a0ffafa253aa6" candidate_uuid: "37d20c04811e464fba5134eb5c296405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ca3576ce0b4b85867b16af0131d7b1"
I20260620 14:20:43.777586 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544a5e3bdc2147c0b30a0ffafa253aa6" candidate_uuid: "37d20c04811e464fba5134eb5c296405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffce03f48b342eba8f55d713a3d9357"
I20260620 14:20:43.777660 4763 raft_consensus.cc:3037] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:43.777726 4630 raft_consensus.cc:3037] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:43.778745 4763 raft_consensus.cc:2454] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d20c04811e464fba5134eb5c296405 in term 1.
I20260620 14:20:43.778919 4630 raft_consensus.cc:2454] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d20c04811e464fba5134eb5c296405 in term 1.
I20260620 14:20:43.779058 4830 leader_election.cc:302] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37d20c04811e464fba5134eb5c296405, 76ca3576ce0b4b85867b16af0131d7b1; no voters:
I20260620 14:20:43.779253 4967 raft_consensus.cc:2781] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:43.779465 4967 raft_consensus.cc:686] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 1 LEADER]: Becoming Leader. State: Replica: 37d20c04811e464fba5134eb5c296405, State: Running, Role: LEADER
I20260620 14:20:43.779855 4967 consensus_queue.cc:237] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } }
I20260620 14:20:43.782837 4485 catalog_manager.cc:5495] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37d20c04811e464fba5134eb5c296405 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "37d20c04811e464fba5134eb5c296405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d20c04811e464fba5134eb5c296405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 } health_report { overall_health: UNKNOWN } } }
I20260620 14:20:43.816905 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.813468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.0.98.3:42967)
0620 14:20:43.813624 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:43.813628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:43.813644 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:20:43.813717 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:43.813720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:43.813783 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:43.813889 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:43.814099 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.814589 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.815387 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:43.815592 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:43.815670 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:43.815996 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:43.816460 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:43.816605 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:43.816660 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:20:43.816721 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:20:43.822881 4763 raft_consensus.cc:1261] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [term 1 FOLLOWER]: Refusing update from remote peer 37d20c04811e464fba5134eb5c296405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:43.822881 4630 raft_consensus.cc:1261] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [term 1 FOLLOWER]: Refusing update from remote peer 37d20c04811e464fba5134eb5c296405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:43.823681 4969 consensus_queue.cc:1035] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ca3576ce0b4b85867b16af0131d7b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:43.823946 4967 consensus_queue.cc:1035] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ffce03f48b342eba8f55d713a3d9357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:43.839089 4485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:43.839280 4485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:43.841701 4485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33c1e5b03494446fa0cdb6ebe0ada5ae: Sending DeleteTablet for 3 replicas of tablet 544a5e3bdc2147c0b30a0ffafa253aa6
I20260620 14:20:43.842383 4876 tablet_service.cc:1514] Processing DeleteTablet for tablet 544a5e3bdc2147c0b30a0ffafa253aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47242
I20260620 14:20:43.842429 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 544a5e3bdc2147c0b30a0ffafa253aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50050
I20260620 14:20:43.842787 4743 tablet_service.cc:1514] Processing DeleteTablet for tablet 544a5e3bdc2147c0b30a0ffafa253aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41762
I20260620 14:20:43.842905 4982 tablet_replica.cc:330] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357: stopping tablet replica
I20260620 14:20:43.843205 4983 tablet_replica.cc:330] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: stopping tablet replica
I20260620 14:20:43.843231 4982 raft_consensus.cc:2229] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:43.843612 4982 raft_consensus.cc:2258] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 1ffce03f48b342eba8f55d713a3d9357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:43.843515 4983 raft_consensus.cc:2229] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:43.844005 4983 raft_consensus.cc:2258] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:43.844044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 4548
I20260620 14:20:43.845624 4983 ts_tablet_manager.cc:1905] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:43.848421 4983 ts_tablet_manager.cc:1918] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:43.848562 4983 log.cc:1196] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/544a5e3bdc2147c0b30a0ffafa253aa6
I20260620 14:20:43.848925 4983 ts_tablet_manager.cc:1939] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 76ca3576ce0b4b85867b16af0131d7b1: Deleting consensus metadata
I20260620 14:20:43.850206 4471 catalog_manager.cc:4841] TS 76ca3576ce0b4b85867b16af0131d7b1 (127.0.98.2:36439): tablet 544a5e3bdc2147c0b30a0ffafa253aa6 (table test-table [id=0dd143f7209f4594a79f8be273cc9d35]) successfully deleted
W20260620 14:20:43.851783 4470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43943 (error 108)
I20260620 14:20:43.851898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 4681
I20260620 14:20:43.852433 4981 tablet_replica.cc:330] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: stopping tablet replica
I20260620 14:20:43.852764 4981 raft_consensus.cc:2229] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:43.853147 4981 raft_consensus.cc:2258] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:43.853425 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:43.853058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43943 (local address 127.0.0.1:50052)
0620 14:20:43.853252 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:43.853323 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260620 14:20:43.853670 4470 catalog_manager.cc:4584] TS 1ffce03f48b342eba8f55d713a3d9357 (127.0.98.1:43943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 544a5e3bdc2147c0b30a0ffafa253aa6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43943: connect: Connection refused (error 111)
I20260620 14:20:43.854616 4981 ts_tablet_manager.cc:1905] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:43.857134 4981 ts_tablet_manager.cc:1918] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:43.857224 4981 log.cc:1196] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/544a5e3bdc2147c0b30a0ffafa253aa6
I20260620 14:20:43.857577 4981 ts_tablet_manager.cc:1939] T 544a5e3bdc2147c0b30a0ffafa253aa6 P 37d20c04811e464fba5134eb5c296405: Deleting consensus metadata
I20260620 14:20:43.858721 4471 catalog_manager.cc:4841] TS 37d20c04811e464fba5134eb5c296405 (127.0.98.3:42967): tablet 544a5e3bdc2147c0b30a0ffafa253aa6 (table test-table [id=0dd143f7209f4594a79f8be273cc9d35]) successfully deleted
I20260620 14:20:43.859380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 4814
I20260620 14:20:43.865150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 4455
2026-06-20T14:20:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2597 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:20:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:43Z Disabled control of system clock
I20260620 14:20:43.894237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44527
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:20:44.003499 4988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:44.003759 4988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:44.003868 4988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:44.007754 4988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:44.007822 4988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:44.007838 4988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:44.007854 4988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:44.007869 4988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:44.012671 4988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:44.013670 4988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:44.014971 4988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:44.023228 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:44.023231 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:44.023288 4988 server_base.cc:1034] running on GCE node
W20260620 14:20:44.023228 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:44.024097 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:44.025183 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:44.026343 4988 hybrid_clock.cc:648] HybridClock initialized: now 1781965244026317 us; error 42 us; skew 500 ppm
I20260620 14:20:44.028177 4988 webserver.cc:460] Webserver started at http://127.0.98.62:39767/ using document root <none> and password file <none>
I20260620 14:20:44.028738 4988 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:44.028800 4988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:44.029024 4988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:44.030998 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "98c5cf63bdd2421c870c3c07f4e22495"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "5fdc68617ef1d61e72757bb7cdd57032"
server_key_iv: "534facca19357d7cbdcbf37772009082"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.031574 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "98c5cf63bdd2421c870c3c07f4e22495"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "5fdc68617ef1d61e72757bb7cdd57032"
server_key_iv: "534facca19357d7cbdcbf37772009082"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.035076 4988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:20:44.037456 5002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.038614 4988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:44.038769 4988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "98c5cf63bdd2421c870c3c07f4e22495"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "5fdc68617ef1d61e72757bb7cdd57032"
server_key_iv: "534facca19357d7cbdcbf37772009082"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.038879 4988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:44.043332 4988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:44.043955 4988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:44.044139 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:44.050841 4988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44527
I20260620 14:20:44.050856 5054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44527 every 8 connection(s)
I20260620 14:20:44.051816 4988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:44.054843 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:44.059306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 4988
I20260620 14:20:44.059459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:44.059736 392 external_mini_cluster.cc:1428] Setting key 75f6424b54dbfc34585f519de7ff5a18
I20260620 14:20:44.060580 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495: Bootstrap starting.
I20260620 14:20:44.062991 5055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:44.063938 5055 log.cc:824] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:44.065419 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.061190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.0.98.62:44527)
0620 14:20:44.061873 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.061888 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.061923 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:20:44.062575 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.062593 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.062948 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.063282 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.064142 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.064150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.064169 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.064180 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.064235 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.064252 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.064378 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.064483 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.064735 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":147,"threads_started":1}
I20260620 14:20:44.065935 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495: No bootstrap required, opened a new log
I20260620 14:20:44.069764 5055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c5cf63bdd2421c870c3c07f4e22495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44527 } }
I20260620 14:20:44.070025 5055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:44.070101 5055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c5cf63bdd2421c870c3c07f4e22495, State: Initialized, Role: FOLLOWER
I20260620 14:20:44.070516 5055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c5cf63bdd2421c870c3c07f4e22495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44527 } }
I20260620 14:20:44.070659 5055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:44.070739 5055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:44.070850 5055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:44.071823 5055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c5cf63bdd2421c870c3c07f4e22495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44527 } }
I20260620 14:20:44.072165 5055 leader_election.cc:302] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c5cf63bdd2421c870c3c07f4e22495; no voters:
I20260620 14:20:44.072504 5055 leader_election.cc:288] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:44.072604 5060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:44.072850 5060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [term 1 LEADER]: Becoming Leader. State: Replica: 98c5cf63bdd2421c870c3c07f4e22495, State: Running, Role: LEADER
I20260620 14:20:44.073689 5055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:44.073710 5060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c5cf63bdd2421c870c3c07f4e22495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44527 } }
I20260620 14:20:44.075443 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98c5cf63bdd2421c870c3c07f4e22495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c5cf63bdd2421c870c3c07f4e22495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44527 } } }
I20260620 14:20:44.075472 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c5cf63bdd2421c870c3c07f4e22495. Latest consensus state: current_term: 1 leader_uuid: "98c5cf63bdd2421c870c3c07f4e22495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c5cf63bdd2421c870c3c07f4e22495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44527 } } }
I20260620 14:20:44.075569 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:44.075572 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495 [sys.catalog]: This master's current role is: LEADER
W20260620 14:20:44.078680 5075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:20:44.078760 5075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:20:44.078868 5074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:44.079587 5074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:44.084848 5074 catalog_manager.cc:1347] Generated new cluster ID: add8ce70db924eadbc59fc2a7b624404
I20260620 14:20:44.084908 5074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:44.097110 5074 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:44.097445 5074 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:44.104302 5074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495: Generated new TSK 0
I20260620 14:20:44.104959 5074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:20:44.110965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44527
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:20:44.220887 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:44.221160 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:44.221197 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:44.225258 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:44.225308 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:44.225395 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:44.230105 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:44.231128 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:44.232594 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:44.242040 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:44.242062 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:44.242040 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:44.242463 5079 server_base.cc:1034] running on GCE node
I20260620 14:20:44.242905 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:44.243501 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:44.244668 5079 hybrid_clock.cc:648] HybridClock initialized: now 1781965244244632 us; error 46 us; skew 500 ppm
I20260620 14:20:44.246452 5079 webserver.cc:460] Webserver started at http://127.0.98.1:40229/ using document root <none> and password file <none>
I20260620 14:20:44.247008 5079 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:44.247069 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:44.247308 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:44.249413 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "acff761141de4a05aaf77f73a544d8c5"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "57623584879d8b0f5e0e34f334d53086"
server_key_iv: "556d26b47972b114b3f6265111b6224f"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.249989 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "acff761141de4a05aaf77f73a544d8c5"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "57623584879d8b0f5e0e34f334d53086"
server_key_iv: "556d26b47972b114b3f6265111b6224f"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.253571 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:20:44.256049 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.257333 5079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:44.257478 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "acff761141de4a05aaf77f73a544d8c5"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "57623584879d8b0f5e0e34f334d53086"
server_key_iv: "556d26b47972b114b3f6265111b6224f"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.257596 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:44.262154 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:44.262739 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:44.262920 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:44.263443 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:44.264394 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:44.264472 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.264534 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:44.264580 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.273005 5079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46669
I20260620 14:20:44.273074 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46669 every 8 connection(s)
I20260620 14:20:44.274008 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:44.276846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 5079
I20260620 14:20:44.276984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:44.277271 392 external_mini_cluster.cc:1428] Setting key 7d481faeadb7a12574241ed91eff1aac
I20260620 14:20:44.279237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44527
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:20:44.280886 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.275941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40551 (local address 127.0.98.62:44527)
0620 14:20:44.276096 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.276100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.277368 (+ 1268us) server_negotiation.cc:407] Connection header received
0620 14:20:44.278922 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.278925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.278967 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.279054 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.280346 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.280349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.280351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.280363 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.280382 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.280390 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.280470 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.280664 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.280718 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:20:44.281927 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.276455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44527 (local address 127.0.98.1:40551)
0620 14:20:44.277214 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.277267 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:44.278721 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:44.279278 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:44.279290 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:44.279418 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:44.280032 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:20:44.280036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:44.280094 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:20:44.280185 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:44.280478 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:44.280490 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:44.280499 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:20:44.280692 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 14:20:44.280914 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":157,"threads_started":1}
I20260620 14:20:44.283427 5207 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44527
I20260620 14:20:44.283656 5207 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:44.284114 5207 heartbeater.cc:511] Master 127.0.98.62:44527 requested a full tablet report, sending...
I20260620 14:20:44.285650 5019 ts_manager.cc:194] Registered new tserver with Master: acff761141de4a05aaf77f73a544d8c5 (127.0.98.1:46669)
W20260620 14:20:44.393728 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:44.393982 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:44.394022 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:44.398033 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:44.398085 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:44.398139 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:44.402756 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:44.403811 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:44.405140 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:44.414507 5215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:44.414512 5218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:44.414556 5216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:44.414748 5210 server_base.cc:1034] running on GCE node
I20260620 14:20:44.415256 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:44.415920 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:44.417093 5210 hybrid_clock.cc:648] HybridClock initialized: now 1781965244417077 us; error 27 us; skew 500 ppm
I20260620 14:20:44.418838 5210 webserver.cc:460] Webserver started at http://127.0.98.2:40651/ using document root <none> and password file <none>
I20260620 14:20:44.419402 5210 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:44.419492 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:44.419731 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:44.421739 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "e0b86ae400e94dcf831222ae44e63b1b"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "86cd44112ff1076e2ece9f19b40b1118"
server_key_iv: "b3c9d218b3e092d3982b10b7c2358d36"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.422287 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e0b86ae400e94dcf831222ae44e63b1b"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "86cd44112ff1076e2ece9f19b40b1118"
server_key_iv: "b3c9d218b3e092d3982b10b7c2358d36"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.425761 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:20:44.428053 5224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.429236 5210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:20:44.429394 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "e0b86ae400e94dcf831222ae44e63b1b"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "86cd44112ff1076e2ece9f19b40b1118"
server_key_iv: "b3c9d218b3e092d3982b10b7c2358d36"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.429514 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:44.434028 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:44.434615 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:44.434790 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:44.435307 5210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:44.436269 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:44.436338 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.436414 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:44.436465 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.445497 5210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35879
I20260620 14:20:44.445557 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35879 every 8 connection(s)
I20260620 14:20:44.446429 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:44.452270 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.448258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37039 (local address 127.0.98.62:44527)
0620 14:20:44.448419 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.448424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.449507 (+ 1083us) server_negotiation.cc:407] Connection header received
0620 14:20:44.450695 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.450698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.450734 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.450810 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.451889 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.451892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.451894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.451906 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.451925 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.451934 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.452012 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.452127 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.452165 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:20:44.453101 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.448712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44527 (local address 127.0.98.2:37039)
0620 14:20:44.449344 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.449385 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:44.450516 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:44.450976 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:44.450982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:44.451069 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:44.451602 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:20:44.451603 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:44.451643 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:20:44.451708 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:44.452009 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:44.452015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:44.452018 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:20:44.452146 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:20:44.452359 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260620 14:20:44.454200 5338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44527
I20260620 14:20:44.454399 5338 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:44.454757 5338 heartbeater.cc:511] Master 127.0.98.62:44527 requested a full tablet report, sending...
I20260620 14:20:44.455667 5019 ts_manager.cc:194] Registered new tserver with Master: e0b86ae400e94dcf831222ae44e63b1b (127.0.98.2:35879)
I20260620 14:20:44.456341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 5210
I20260620 14:20:44.456493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:44.456768 392 external_mini_cluster.cc:1428] Setting key ace76e3b05db2d4404e4b5339e213b32
I20260620 14:20:44.458477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44527
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:20:44.569621 5341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:44.569917 5341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:44.569958 5341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:44.574151 5341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:44.574244 5341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:44.574312 5341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:44.579493 5341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:44.580574 5341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:44.581882 5341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:44.591485 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:44.591492 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:44.591564 5341 server_base.cc:1034] running on GCE node
W20260620 14:20:44.591526 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:44.592079 5341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:44.592712 5341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:44.593897 5341 hybrid_clock.cc:648] HybridClock initialized: now 1781965244593883 us; error 26 us; skew 500 ppm
I20260620 14:20:44.595758 5341 webserver.cc:460] Webserver started at http://127.0.98.3:32853/ using document root <none> and password file <none>
I20260620 14:20:44.596405 5341 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:44.596501 5341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:44.596730 5341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:44.598728 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "a2f4fd3e3380481f8213597982f661f1"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "b3a806a719ef6273552e86c5e3a8b232"
server_key_iv: "6befac1c42ce8047fce8a5a63387704f"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.599303 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a2f4fd3e3380481f8213597982f661f1"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "b3a806a719ef6273552e86c5e3a8b232"
server_key_iv: "6befac1c42ce8047fce8a5a63387704f"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.602933 5341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:20:44.605647 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.606930 5341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:44.607065 5341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "a2f4fd3e3380481f8213597982f661f1"
format_stamp: "Formatted at 2026-06-20 14:20:44 on dist-test-slave-87nc"
server_key: "b3a806a719ef6273552e86c5e3a8b232"
server_key_iv: "6befac1c42ce8047fce8a5a63387704f"
server_key_version: "encryptionkey@0"
I20260620 14:20:44.607184 5341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:44.611935 5341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:44.612540 5341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:44.612718 5341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:44.613282 5341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:44.614190 5341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:44.614259 5341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.614322 5341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:44.614351 5341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:44.623570 5341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38135
I20260620 14:20:44.623598 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38135 every 8 connection(s)
I20260620 14:20:44.624667 5341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:44.632704 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.627043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40155 (local address 127.0.98.62:44527)
0620 14:20:44.627229 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.627234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.628726 (+ 1492us) server_negotiation.cc:407] Connection header received
0620 14:20:44.630594 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.630600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.630645 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.630744 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.632149 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.632169 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.632174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.632187 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.632210 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.632220 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.632363 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.632518 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.632563 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:20:44.633617 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.627589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44527 (local address 127.0.98.3:40155)
0620 14:20:44.628555 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.628617 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:44.630347 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:44.630974 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:44.630984 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:44.631115 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:44.631841 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:20:44.631844 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:44.631898 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:20:44.631989 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:44.632356 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:44.632365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:44.632372 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:20:44.632561 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 14:20:44.632811 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":200,"threads_started":1}
I20260620 14:20:44.634728 5469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44527
I20260620 14:20:44.634994 5469 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:44.634989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 5341
I20260620 14:20:44.635098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:44.635429 392 external_mini_cluster.cc:1428] Setting key 99822c8d33c548597f04acefc9829818
I20260620 14:20:44.635495 5469 heartbeater.cc:511] Master 127.0.98.62:44527 requested a full tablet report, sending...
I20260620 14:20:44.636818 5018 ts_manager.cc:194] Registered new tserver with Master: a2f4fd3e3380481f8213597982f661f1 (127.0.98.3:38135)
I20260620 14:20:44.638020 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:20:44.641376 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.639896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.0.98.62:44527)
0620 14:20:44.640022 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.640025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.640234 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:20:44.640354 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.640356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.640383 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.640449 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.641073 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.641076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.641078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.641087 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.641104 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.641112 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.641177 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.641248 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.641278 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:20:44.644253 5018 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:44.647150 5018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:44.658761 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.655272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38135 (local address 127.0.0.1:41134)
0620 14:20:44.656473 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.656484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:44.656575 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:44.657545 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:44.657548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:44.657563 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:44.658014 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:20:44.658015 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:44.658030 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:20:44.658080 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:44.658427 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:44.658435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:44.658441 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:20:44.658548 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:20:44.658617 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":89,"threads_started":1}
I20260620 14:20:44.659097 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.655281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.0.98.1:46669)
0620 14:20:44.656602 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.656613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.656632 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:20:44.656681 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.656689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.656872 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.657016 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.658284 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.658289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.658298 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.658308 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.658344 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.658364 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.658495 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.658831 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.658908 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1209,"thread_start_us":91,"threads_started":1}
I20260620 14:20:44.659322 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.655932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.0.98.3:38135)
0620 14:20:44.656343 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.656357 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.656571 (+ 214us) server_negotiation.cc:407] Connection header received
0620 14:20:44.656661 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.656667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.656862 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.657023 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.658237 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.658243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.658256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.658267 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.658310 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.658334 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.658441 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.659066 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.659140 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":137,"threads_started":1}
I20260620 14:20:44.659363 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.655171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46669 (local address 127.0.0.1:52636)
0620 14:20:44.656147 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.656195 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:44.656287 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:44.657013 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:44.657018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:44.657091 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:44.658090 (+ 999us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:20:44.658092 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:44.658104 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:20:44.658137 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:44.658582 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:44.658585 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:44.658589 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:20:44.658663 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:20:44.658697 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":78,"threads_started":1}
I20260620 14:20:44.659451 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.655932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.0.98.2:35879)
0620 14:20:44.656387 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.656399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.656598 (+ 199us) server_negotiation.cc:407] Connection header received
0620 14:20:44.656766 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.656772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.656923 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.657032 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.658665 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.658671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.658681 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.658692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.658743 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.658780 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.658865 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.659215 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.659282 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":111,"threads_started":1}
I20260620 14:20:44.659801 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.655523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35879 (local address 127.0.0.1:37710)
0620 14:20:44.656501 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.656515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:44.656661 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:44.657723 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:44.657728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:44.657744 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:44.658498 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:20:44.658502 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:44.658514 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:20:44.658540 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:44.658935 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:44.658938 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:44.658940 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:20:44.659629 (+ 689us) client_negotiation.cc:240] Negotiation successful
0620 14:20:44.659663 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":82,"threads_started":1}
I20260620 14:20:44.662223 5272 tablet_service.cc:1467] Processing CreateTablet for tablet fdfc32dd598e42c09821f4eeeebf2383 (DEFAULT_TABLE table=test-table [id=928cb5abcc63437ea0c567c8c1a80f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:44.662212 5403 tablet_service.cc:1467] Processing CreateTablet for tablet fdfc32dd598e42c09821f4eeeebf2383 (DEFAULT_TABLE table=test-table [id=928cb5abcc63437ea0c567c8c1a80f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:44.662212 5141 tablet_service.cc:1467] Processing CreateTablet for tablet fdfc32dd598e42c09821f4eeeebf2383 (DEFAULT_TABLE table=test-table [id=928cb5abcc63437ea0c567c8c1a80f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:44.663264 5272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdfc32dd598e42c09821f4eeeebf2383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:44.663271 5403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdfc32dd598e42c09821f4eeeebf2383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:44.663271 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdfc32dd598e42c09821f4eeeebf2383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:44.669179 5486 tablet_bootstrap.cc:492] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b: Bootstrap starting.
I20260620 14:20:44.670619 5488 tablet_bootstrap.cc:492] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: Bootstrap starting.
I20260620 14:20:44.671590 5486 tablet_bootstrap.cc:654] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:44.672075 5487 tablet_bootstrap.cc:492] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5: Bootstrap starting.
I20260620 14:20:44.672441 5486 log.cc:824] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:44.672611 5488 tablet_bootstrap.cc:654] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:44.673424 5488 log.cc:824] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:44.674160 5487 tablet_bootstrap.cc:654] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:44.674444 5486 tablet_bootstrap.cc:492] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b: No bootstrap required, opened a new log
I20260620 14:20:44.674685 5486 ts_tablet_manager.cc:1397] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:20:44.674908 5487 log.cc:824] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:44.675525 5488 tablet_bootstrap.cc:492] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: No bootstrap required, opened a new log
I20260620 14:20:44.675719 5488 ts_tablet_manager.cc:1397] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:20:44.676800 5487 tablet_bootstrap.cc:492] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5: No bootstrap required, opened a new log
I20260620 14:20:44.676975 5487 ts_tablet_manager.cc:1397] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:20:44.679173 5488 raft_consensus.cc:348] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } }
I20260620 14:20:44.679428 5488 raft_consensus.cc:374] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:44.679289 5486 raft_consensus.cc:348] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } }
I20260620 14:20:44.679488 5488 raft_consensus.cc:729] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f4fd3e3380481f8213597982f661f1, State: Initialized, Role: FOLLOWER
I20260620 14:20:44.679541 5486 raft_consensus.cc:374] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:44.679616 5486 raft_consensus.cc:729] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b86ae400e94dcf831222ae44e63b1b, State: Initialized, Role: FOLLOWER
I20260620 14:20:44.680035 5488 consensus_queue.cc:260] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } }
I20260620 14:20:44.680120 5486 consensus_queue.cc:260] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } }
I20260620 14:20:44.680887 5488 ts_tablet_manager.cc:1428] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260620 14:20:44.681018 5486 ts_tablet_manager.cc:1428] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:20:44.681260 5469 heartbeater.cc:503] Master 127.0.98.62:44527 was elected leader, sending a full tablet report...
I20260620 14:20:44.681332 5338 heartbeater.cc:503] Master 127.0.98.62:44527 was elected leader, sending a full tablet report...
I20260620 14:20:44.681340 5487 raft_consensus.cc:348] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } }
I20260620 14:20:44.681555 5487 raft_consensus.cc:374] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:44.681602 5487 raft_consensus.cc:729] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acff761141de4a05aaf77f73a544d8c5, State: Initialized, Role: FOLLOWER
I20260620 14:20:44.681974 5487 consensus_queue.cc:260] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } }
I20260620 14:20:44.682571 5207 heartbeater.cc:503] Master 127.0.98.62:44527 was elected leader, sending a full tablet report...
I20260620 14:20:44.682830 5487 ts_tablet_manager.cc:1428] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 14:20:44.697357 5339 tablet.cc:2369] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:44.775185 5208 tablet.cc:2369] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:44.876988 5470 tablet.cc:2369] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:44.881754 5493 raft_consensus.cc:482] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:44.881955 5493 raft_consensus.cc:504] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } }
I20260620 14:20:44.883164 5493 leader_election.cc:288] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acff761141de4a05aaf77f73a544d8c5 (127.0.98.1:46669), a2f4fd3e3380481f8213597982f661f1 (127.0.98.3:38135)
I20260620 14:20:44.885138 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.883387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46669 (local address 127.0.98.2:38785)
0620 14:20:44.883524 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.883540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:44.883628 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:44.883976 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:44.883978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:44.883993 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:44.884516 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:20:44.884519 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:44.884533 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:20:44.884566 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:44.884866 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:44.884869 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:44.884873 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:20:44.884963 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:20:44.885002 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:20:44.885175 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.883590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38785 (local address 127.0.98.1:46669)
0620 14:20:44.883724 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.883729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.883743 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:20:44.883836 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.883839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.883871 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.883958 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.884716 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.884719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.884722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.884733 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.884755 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.884767 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.884855 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.884991 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.885035 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:20:44.885274 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.883547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56619 (local address 127.0.98.3:38135)
0620 14:20:44.883683 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.883688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.884007 (+ 319us) server_negotiation.cc:407] Connection header received
0620 14:20:44.884134 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.884137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.884175 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.884258 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.884866 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.884870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.884873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.884886 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.884910 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.884924 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.885010 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.885090 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.885125 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:20:44.885654 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.883486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38135 (local address 127.0.98.2:56619)
0620 14:20:44.883904 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.883915 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:44.884021 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:44.884272 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:44.884275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:44.884288 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:44.884686 (+ 398us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:20:44.884687 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:44.884698 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:20:44.884721 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:44.885008 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:44.885011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:44.885015 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:20:44.885091 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:20:44.885122 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":128,"threads_started":1}
I20260620 14:20:44.885824 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdfc32dd598e42c09821f4eeeebf2383" candidate_uuid: "e0b86ae400e94dcf831222ae44e63b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acff761141de4a05aaf77f73a544d8c5" is_pre_election: true
I20260620 14:20:44.886152 5161 raft_consensus.cc:2454] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b86ae400e94dcf831222ae44e63b1b in term 0.
I20260620 14:20:44.886266 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdfc32dd598e42c09821f4eeeebf2383" candidate_uuid: "e0b86ae400e94dcf831222ae44e63b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f4fd3e3380481f8213597982f661f1" is_pre_election: true
I20260620 14:20:44.886547 5423 raft_consensus.cc:2454] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b86ae400e94dcf831222ae44e63b1b in term 0.
I20260620 14:20:44.886641 5227 leader_election.cc:302] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acff761141de4a05aaf77f73a544d8c5, e0b86ae400e94dcf831222ae44e63b1b; no voters:
I20260620 14:20:44.886938 5493 raft_consensus.cc:2781] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:44.887025 5493 raft_consensus.cc:482] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:44.887073 5493 raft_consensus.cc:3037] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:44.888266 5493 raft_consensus.cc:504] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } }
I20260620 14:20:44.888628 5493 leader_election.cc:288] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [CANDIDATE]: Term 1 election: Requested vote from peers acff761141de4a05aaf77f73a544d8c5 (127.0.98.1:46669), a2f4fd3e3380481f8213597982f661f1 (127.0.98.3:38135)
I20260620 14:20:44.888952 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdfc32dd598e42c09821f4eeeebf2383" candidate_uuid: "e0b86ae400e94dcf831222ae44e63b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f4fd3e3380481f8213597982f661f1"
I20260620 14:20:44.888952 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdfc32dd598e42c09821f4eeeebf2383" candidate_uuid: "e0b86ae400e94dcf831222ae44e63b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acff761141de4a05aaf77f73a544d8c5"
I20260620 14:20:44.889048 5423 raft_consensus.cc:3037] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:44.889083 5161 raft_consensus.cc:3037] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:44.890023 5423 raft_consensus.cc:2454] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b86ae400e94dcf831222ae44e63b1b in term 1.
I20260620 14:20:44.890134 5161 raft_consensus.cc:2454] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b86ae400e94dcf831222ae44e63b1b in term 1.
I20260620 14:20:44.890288 5226 leader_election.cc:302] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f4fd3e3380481f8213597982f661f1, e0b86ae400e94dcf831222ae44e63b1b; no voters:
I20260620 14:20:44.890497 5493 raft_consensus.cc:2781] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:44.890717 5493 raft_consensus.cc:686] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 1 LEADER]: Becoming Leader. State: Replica: e0b86ae400e94dcf831222ae44e63b1b, State: Running, Role: LEADER
I20260620 14:20:44.891042 5493 consensus_queue.cc:237] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } }
I20260620 14:20:44.894250 5017 catalog_manager.cc:5495] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b86ae400e94dcf831222ae44e63b1b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e0b86ae400e94dcf831222ae44e63b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b86ae400e94dcf831222ae44e63b1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35879 } health_report { overall_health: HEALTHY } } }
I20260620 14:20:44.950156 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.948838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.0.98.2:35879)
0620 14:20:44.948972 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:44.948976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:44.948991 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:44.949068 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:44.949070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:44.949131 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:44.949211 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:44.949833 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:44.949836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:44.949838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:44.949850 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:44.949869 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:44.949877 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:44.949963 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:44.950016 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:20:44.950051 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:20:44.956178 5161 raft_consensus.cc:1261] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5 [term 1 FOLLOWER]: Refusing update from remote peer e0b86ae400e94dcf831222ae44e63b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:44.956457 5423 raft_consensus.cc:1261] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [term 1 FOLLOWER]: Refusing update from remote peer e0b86ae400e94dcf831222ae44e63b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:44.956983 5493 consensus_queue.cc:1035] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "acff761141de4a05aaf77f73a544d8c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:44.957218 5496 consensus_queue.cc:1035] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f4fd3e3380481f8213597982f661f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:44.965879 5499 mvcc.cc:205] Tried to move back new op lower bound from 7298929643332976640 to 7298929643076034560. Current Snapshot: MvccSnapshot[applied={T|T < 7298929643332976640}]
I20260620 14:20:44.966085 5501 mvcc.cc:205] Tried to move back new op lower bound from 7298929643332976640 to 7298929643076034560. Current Snapshot: MvccSnapshot[applied={T|T < 7298929643332976640}]
I20260620 14:20:44.967303 5500 mvcc.cc:205] Tried to move back new op lower bound from 7298929643332976640 to 7298929643076034560. Current Snapshot: MvccSnapshot[applied={T|T < 7298929643332976640}]
I20260620 14:20:44.976457 5018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:44.976650 5018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:44.979127 5018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98c5cf63bdd2421c870c3c07f4e22495: Sending DeleteTablet for 3 replicas of tablet fdfc32dd598e42c09821f4eeeebf2383
I20260620 14:20:44.980001 5403 tablet_service.cc:1514] Processing DeleteTablet for tablet fdfc32dd598e42c09821f4eeeebf2383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:44 UTC) from {username='slave'} at 127.0.0.1:41134
I20260620 14:20:44.980144 5272 tablet_service.cc:1514] Processing DeleteTablet for tablet fdfc32dd598e42c09821f4eeeebf2383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:44 UTC) from {username='slave'} at 127.0.0.1:37710
I20260620 14:20:44.980556 5509 tablet_replica.cc:330] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: stopping tablet replica
I20260620 14:20:44.980587 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet fdfc32dd598e42c09821f4eeeebf2383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:44 UTC) from {username='slave'} at 127.0.0.1:52636
I20260620 14:20:44.980911 5509 raft_consensus.cc:2229] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:44.981117 5511 tablet_replica.cc:330] T fdfc32dd598e42c09821f4eeeebf2383 P acff761141de4a05aaf77f73a544d8c5: stopping tablet replica
I20260620 14:20:44.981251 5509 raft_consensus.cc:2258] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:44.981537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 5079
I20260620 14:20:44.982973 5509 ts_tablet_manager.cc:1905] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:44.985846 5509 ts_tablet_manager.cc:1918] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:44.985963 5509 log.cc:1196] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/fdfc32dd598e42c09821f4eeeebf2383
I20260620 14:20:44.986265 5509 ts_tablet_manager.cc:1939] T fdfc32dd598e42c09821f4eeeebf2383 P a2f4fd3e3380481f8213597982f661f1: Deleting consensus metadata
I20260620 14:20:44.987537 5004 catalog_manager.cc:4841] TS a2f4fd3e3380481f8213597982f661f1 (127.0.98.3:38135): tablet fdfc32dd598e42c09821f4eeeebf2383 (table test-table [id=928cb5abcc63437ea0c567c8c1a80f34]) successfully deleted
I20260620 14:20:44.988015 5510 tablet_replica.cc:330] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b: stopping tablet replica
I20260620 14:20:44.988370 5510 raft_consensus.cc:2229] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:44.988770 5510 raft_consensus.cc:2258] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:44.990101 5510 ts_tablet_manager.cc:1905] T fdfc32dd598e42c09821f4eeeebf2383 P e0b86ae400e94dcf831222ae44e63b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:20:44.990420 5005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46669 (error 108)
I20260620 14:20:44.991067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 5210
I20260620 14:20:44.996766 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.996425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46669 (local address 127.0.0.1:52648)
0620 14:20:44.996557 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.996665 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:20:44.997016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 5341
W20260620 14:20:44.997033 5005 catalog_manager.cc:4584] TS acff761141de4a05aaf77f73a544d8c5 (127.0.98.1:46669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fdfc32dd598e42c09821f4eeeebf2383: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46669: connect: Connection refused (error 111)
I20260620 14:20:44.998754 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:44.997109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35879 (local address 127.0.0.1:37726)
0620 14:20:44.998648 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:44.998692 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1473}
I20260620 14:20:45.002436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 4988
2026-06-20T14:20:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1137 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:20:45.024008 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:20:45.025609 392 webserver.cc:460] Webserver started at https://127.0.0.1:34083/ using document root <none> and password file <none>
I20260620 14:20:45.025822 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:20:45.026106 392 webserver.cc:460] Webserver started at http://127.0.0.1:33721/ using document root <none> and password file <none>
2026-06-20T14:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:45Z Disabled control of system clock
I20260620 14:20:45.034852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42073
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42073
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34083/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/testchainca.cert with env {}
W20260620 14:20:45.148428 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:45.148753 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:45.148829 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:45.153456 5518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:45.153550 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:45.153579 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:45.153621 5518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:45.153656 5518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:20:45.153692 5518 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:20:45.153721 5518 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34083/jwks.json
I20260620 14:20:45.159243 5518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34083/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:45.160449 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:45.161521 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:45.170274 5523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:45.170277 5524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:45.170248 5526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:45.170413 5518 server_base.cc:1034] running on GCE node
I20260620 14:20:45.171147 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:45.172313 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:45.173472 5518 hybrid_clock.cc:648] HybridClock initialized: now 1781965245173461 us; error 28 us; skew 500 ppm
I20260620 14:20:45.175330 5518 webserver.cc:460] Webserver started at http://127.0.98.62:43063/ using document root <none> and password file <none>
I20260620 14:20:45.175968 5518 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:45.176043 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:45.176287 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:45.178313 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "0f6022e664d344b494af7f9e30472f4a"
format_stamp: "Formatted at 2026-06-20 14:20:45 on dist-test-slave-87nc"
server_key: "984b5ccad006d83f18c27bc7b5dbc122"
server_key_iv: "45942c2d16ef5bb966c4102aff9cd77e"
server_key_version: "encryptionkey@0"
I20260620 14:20:45.178898 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f6022e664d344b494af7f9e30472f4a"
format_stamp: "Formatted at 2026-06-20 14:20:45 on dist-test-slave-87nc"
server_key: "984b5ccad006d83f18c27bc7b5dbc122"
server_key_iv: "45942c2d16ef5bb966c4102aff9cd77e"
server_key_version: "encryptionkey@0"
I20260620 14:20:45.182535 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:20:45.185027 5532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:45.186314 5518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:45.186491 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "0f6022e664d344b494af7f9e30472f4a"
format_stamp: "Formatted at 2026-06-20 14:20:45 on dist-test-slave-87nc"
server_key: "984b5ccad006d83f18c27bc7b5dbc122"
server_key_iv: "45942c2d16ef5bb966c4102aff9cd77e"
server_key_version: "encryptionkey@0"
I20260620 14:20:45.186612 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:45.204370 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:45.205120 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:45.205363 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:45.212344 5518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42073
I20260620 14:20:45.212357 5584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42073 every 8 connection(s)
I20260620 14:20:45.213310 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:45.216477 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:45.220537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 5518
I20260620 14:20:45.220693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:45.220995 392 external_mini_cluster.cc:1428] Setting key b26176e0fa2cf21532e851ed9ff1eb08
I20260620 14:20:45.222111 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a: Bootstrap starting.
I20260620 14:20:45.224509 5585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:45.225237 5585 log.cc:824] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:45.227114 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a: No bootstrap required, opened a new log
I20260620 14:20:45.230611 5585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6022e664d344b494af7f9e30472f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42073 } }
I20260620 14:20:45.230851 5585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:45.230937 5585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6022e664d344b494af7f9e30472f4a, State: Initialized, Role: FOLLOWER
I20260620 14:20:45.231217 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:45.222480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.0.98.62:42073)
0620 14:20:45.223146 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:45.223161 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:45.223197 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:20:45.224001 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:45.224020 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:45.224449 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:45.224856 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:45.225356 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.226510 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.227568 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.228154 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.228381 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:45.229282 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:45.229288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:45.229301 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:45.229708 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:45.229768 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:45.229788 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:45.229903 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:45.230047 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:20:45.230320 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260620 14:20:45.231427 5585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6022e664d344b494af7f9e30472f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42073 } }
I20260620 14:20:45.231603 5585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:45.231660 5585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:45.231742 5585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:45.232745 5585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6022e664d344b494af7f9e30472f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42073 } }
I20260620 14:20:45.233125 5585 leader_election.cc:302] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f6022e664d344b494af7f9e30472f4a; no voters:
I20260620 14:20:45.233482 5585 leader_election.cc:288] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:45.233577 5590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:45.233767 5590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [term 1 LEADER]: Becoming Leader. State: Replica: 0f6022e664d344b494af7f9e30472f4a, State: Running, Role: LEADER
I20260620 14:20:45.234072 5590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6022e664d344b494af7f9e30472f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42073 } }
I20260620 14:20:45.234350 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:20:45.234752 5585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:45.235729 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f6022e664d344b494af7f9e30472f4a. Latest consensus state: current_term: 1 leader_uuid: "0f6022e664d344b494af7f9e30472f4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6022e664d344b494af7f9e30472f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42073 } } }
I20260620 14:20:45.235880 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:45.236297 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f6022e664d344b494af7f9e30472f4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6022e664d344b494af7f9e30472f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42073 } } }
I20260620 14:20:45.236454 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:45.236810 5599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:45.239588 5599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:45.245167 5599 catalog_manager.cc:1347] Generated new cluster ID: 377f78ff4d7c4ba9a156771951876cfa
I20260620 14:20:45.245249 5599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:45.260397 5599 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:45.261701 5599 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:45.269534 5599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f6022e664d344b494af7f9e30472f4a: Generated new TSK 0
I20260620 14:20:45.270347 5599 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:20:45.307909 5615 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:20:45.308032 5615 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:20:45.308614 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:45.284514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.0.98.62:42073)
0620 14:20:45.284666 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:45.284674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:45.284959 (+ 285us) server_negotiation.cc:407] Connection header received
0620 14:20:45.285195 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:45.285198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:45.285250 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:45.285359 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:20:45.285632 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.286171 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.287088 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.287322 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.287437 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:45.287838 (+ 401us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:20:45.308104 (+ 20266us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:20:45.308454 (+ 350us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 14:20:45.308713 5588 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:20:45.309829 5614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:45.284430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42073 (local address 127.0.0.1:44588)
0620 14:20:45.284861 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:45.284873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:45.285023 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:45.285374 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:45.285377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:45.285390 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:20:45.285515 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:45.285521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.286304 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.286310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:45.286952 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:45.286960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.287431 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.287454 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:45.287551 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:45.287592 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:20:45.308379 (+ 20787us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:20:45.308632 (+ 253us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42073: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42073: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:20:45.310544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 5518
2026-06-20T14:20:45Z chronyd exiting
I20260620 14:20:45.348039 392 test_util.cc:175] -----------------------------------------------
I20260620 14:20:45.348130 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965214882667-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:20:45.349138 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:20:45.350013 392 webserver.cc:460] Webserver started at https://127.0.0.1:36983/ using document root <none> and password file <none>
I20260620 14:20:45.350136 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:20:45.350450 392 webserver.cc:460] Webserver started at http://127.0.0.1:41725/ using document root <none> and password file <none>
2026-06-20T14:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:45Z Disabled control of system clock
I20260620 14:20:45.358830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39197
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39197
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36983/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/testchainca.cert with env {}
W20260620 14:20:45.481179 5621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:45.481534 5621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:45.481616 5621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:45.485777 5621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:45.485865 5621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:45.485903 5621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:45.485939 5621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:45.485970 5621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:20:45.486006 5621 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:20:45.486038 5621 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36983/jwks.json
I20260620 14:20:45.491485 5621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36983/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:45.492628 5621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:45.493717 5621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:45.502125 5627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:45.502239 5621 server_base.cc:1034] running on GCE node
W20260620 14:20:45.502146 5626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:45.502149 5629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:45.503011 5621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:45.504137 5621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:45.505298 5621 hybrid_clock.cc:648] HybridClock initialized: now 1781965245505285 us; error 27 us; skew 500 ppm
I20260620 14:20:45.507055 5621 webserver.cc:460] Webserver started at http://127.0.98.62:45773/ using document root <none> and password file <none>
I20260620 14:20:45.507628 5621 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:45.507691 5621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:45.507969 5621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:45.510001 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "6b8773b25d9b454bbbeffea8563a9643"
format_stamp: "Formatted at 2026-06-20 14:20:45 on dist-test-slave-87nc"
server_key: "6ccb37b4e448ff379d56bfa5a2eb763d"
server_key_iv: "1ea95cba2e94f5b850f9ff1ddf094f1b"
server_key_version: "encryptionkey@0"
I20260620 14:20:45.510569 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b8773b25d9b454bbbeffea8563a9643"
format_stamp: "Formatted at 2026-06-20 14:20:45 on dist-test-slave-87nc"
server_key: "6ccb37b4e448ff379d56bfa5a2eb763d"
server_key_iv: "1ea95cba2e94f5b850f9ff1ddf094f1b"
server_key_version: "encryptionkey@0"
I20260620 14:20:45.514262 5621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:20:45.516610 5635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:45.517839 5621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:45.517982 5621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "6b8773b25d9b454bbbeffea8563a9643"
format_stamp: "Formatted at 2026-06-20 14:20:45 on dist-test-slave-87nc"
server_key: "6ccb37b4e448ff379d56bfa5a2eb763d"
server_key_iv: "1ea95cba2e94f5b850f9ff1ddf094f1b"
server_key_version: "encryptionkey@0"
I20260620 14:20:45.518095 5621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:45.538892 5621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:45.539681 5621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:45.539917 5621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:45.546839 5621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39197
I20260620 14:20:45.546844 5687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39197 every 8 connection(s)
I20260620 14:20:45.548142 5621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:45.551272 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:45.554525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 5621
I20260620 14:20:45.554658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:45.554965 392 external_mini_cluster.cc:1428] Setting key 46e11d9ece62d51db77c958f88c15c17
I20260620 14:20:45.557420 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643: Bootstrap starting.
I20260620 14:20:45.559631 5688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:45.560395 5688 log.cc:824] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:45.562353 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643: No bootstrap required, opened a new log
I20260620 14:20:45.564417 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:45.556372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.0.98.62:39197)
0620 14:20:45.557062 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:45.557083 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:45.557121 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:20:45.557842 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:45.557862 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:45.558292 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:45.558647 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:45.559089 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.560173 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.561118 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.561588 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.561760 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:45.562678 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:45.562683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:45.562697 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:45.563090 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:45.563145 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:45.563165 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:45.563277 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:45.563409 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:20:45.563660 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":132,"threads_started":1}
I20260620 14:20:45.566198 5688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8773b25d9b454bbbeffea8563a9643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } }
I20260620 14:20:45.566474 5688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:45.566550 5688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8773b25d9b454bbbeffea8563a9643, State: Initialized, Role: FOLLOWER
I20260620 14:20:45.567020 5688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8773b25d9b454bbbeffea8563a9643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } }
I20260620 14:20:45.567140 5688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:45.567216 5688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:45.567318 5688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:45.567431 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:20:45.568425 5688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8773b25d9b454bbbeffea8563a9643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } }
I20260620 14:20:45.568755 5688 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b8773b25d9b454bbbeffea8563a9643; no voters:
I20260620 14:20:45.569118 5688 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:45.569233 5693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:45.569478 5693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [term 1 LEADER]: Becoming Leader. State: Replica: 6b8773b25d9b454bbbeffea8563a9643, State: Running, Role: LEADER
I20260620 14:20:45.569796 5693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8773b25d9b454bbbeffea8563a9643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } }
I20260620 14:20:45.570356 5688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:45.571374 5694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b8773b25d9b454bbbeffea8563a9643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8773b25d9b454bbbeffea8563a9643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } } }
I20260620 14:20:45.571525 5694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:45.571821 5695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8773b25d9b454bbbeffea8563a9643. Latest consensus state: current_term: 1 leader_uuid: "6b8773b25d9b454bbbeffea8563a9643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8773b25d9b454bbbeffea8563a9643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39197 } } }
I20260620 14:20:45.571910 5695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:45.572024 5702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:45.575297 5702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:45.581015 5702 catalog_manager.cc:1347] Generated new cluster ID: 28ab4a3a6052484781e33959c0aa6a32
I20260620 14:20:45.581084 5702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:45.587853 5702 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:45.588722 5702 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:45.595981 5702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b8773b25d9b454bbbeffea8563a9643: Generated new TSK 0
I20260620 14:20:45.596638 5702 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:20:45.628661 5718 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:20:45.628762 5718 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:20:45.629361 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:45.606259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.0.98.62:39197)
0620 14:20:45.606391 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:45.606396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:45.606571 (+ 175us) server_negotiation.cc:407] Connection header received
0620 14:20:45.606721 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:45.606724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:45.606768 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:45.606857 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:20:45.607117 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.607625 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.608858 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.609080 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.609158 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:45.609464 (+ 306us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:20:45.629002 (+ 19538us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:20:45.629245 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 14:20:45.629456 5691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:20:45.629590 5717 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:45.606194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39197 (local address 127.0.0.1:46482)
0620 14:20:45.606482 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:45.606497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:45.606608 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:45.606880 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:45.606883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:45.606898 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:20:45.607015 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:45.607021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.607745 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.607748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:45.608749 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:45.608759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.609174 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.609198 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:45.609285 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:45.609314 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:20:45.629223 (+ 19909us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:20:45.629416 (+ 193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39197: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36983/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
I20260620 14:20:45.630349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 5621
2026-06-20T14:20:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (320 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:20:45.669945 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:20:45.670855 392 webserver.cc:460] Webserver started at https://127.0.0.1:35691/ using document root <none> and password file <none>
I20260620 14:20:45.671008 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:20:45.671294 392 webserver.cc:460] Webserver started at http://127.0.0.1:46491/ using document root <none> and password file <none>
2026-06-20T14:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:45Z Disabled control of system clock
I20260620 14:20:45.680368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43303
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43303
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35691/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:20:45.792382 5724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:45.792701 5724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:45.792801 5724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:45.797019 5724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:45.797094 5724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:45.797111 5724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:45.797128 5724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:45.797197 5724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:20:45.797227 5724 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:20:45.797260 5724 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35691/jwks.json
I20260620 14:20:45.802891 5724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35691/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:45.804126 5724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:45.805615 5724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:45.814550 5732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:45.814540 5730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:45.814540 5729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:45.814901 5724 server_base.cc:1034] running on GCE node
I20260620 14:20:45.815483 5724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:45.816744 5724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:45.817950 5724 hybrid_clock.cc:648] HybridClock initialized: now 1781965245817936 us; error 30 us; skew 500 ppm
I20260620 14:20:45.820021 5724 webserver.cc:460] Webserver started at http://127.0.98.62:43861/ using document root <none> and password file <none>
I20260620 14:20:45.820640 5724 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:45.820741 5724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:45.820996 5724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:45.823192 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "42ec93e8e482459391c9890a727d49df"
format_stamp: "Formatted at 2026-06-20 14:20:45 on dist-test-slave-87nc"
server_key: "42e7a55957ba478c57fb169520202cbb"
server_key_iv: "3aa82ebb00cc2db58b4c69fe22e2a3fa"
server_key_version: "encryptionkey@0"
I20260620 14:20:45.823890 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "42ec93e8e482459391c9890a727d49df"
format_stamp: "Formatted at 2026-06-20 14:20:45 on dist-test-slave-87nc"
server_key: "42e7a55957ba478c57fb169520202cbb"
server_key_iv: "3aa82ebb00cc2db58b4c69fe22e2a3fa"
server_key_version: "encryptionkey@0"
I20260620 14:20:45.827740 5724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:20:45.830489 5738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:45.831857 5724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:20:45.832023 5724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "42ec93e8e482459391c9890a727d49df"
format_stamp: "Formatted at 2026-06-20 14:20:45 on dist-test-slave-87nc"
server_key: "42e7a55957ba478c57fb169520202cbb"
server_key_iv: "3aa82ebb00cc2db58b4c69fe22e2a3fa"
server_key_version: "encryptionkey@0"
I20260620 14:20:45.832154 5724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:45.853668 5724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:45.854439 5724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:45.854660 5724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:45.861706 5724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43303
I20260620 14:20:45.861716 5790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43303 every 8 connection(s)
I20260620 14:20:45.862690 5724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:45.866001 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:45.866516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 5724
I20260620 14:20:45.866636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:45.866957 392 external_mini_cluster.cc:1428] Setting key 68cd8f737d906da67dd13cbf0a0a0691
I20260620 14:20:45.873361 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df: Bootstrap starting.
I20260620 14:20:45.875648 5791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:45.876101 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:45.868252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.0.98.62:43303)
0620 14:20:45.868866 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:45.868880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:45.868920 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:20:45.869605 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:45.869624 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:45.870028 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:45.870371 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:45.870794 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.871952 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.872832 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.873305 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.873505 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:45.874400 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:45.874406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:45.874418 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:45.874789 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:45.874844 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:45.874863 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:45.874971 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:45.875083 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:20:45.875331 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
I20260620 14:20:45.876482 5791 log.cc:824] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:45.878479 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df: No bootstrap required, opened a new log
I20260620 14:20:45.878693 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:20:45.881856 5791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ec93e8e482459391c9890a727d49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } }
I20260620 14:20:45.882102 5791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:45.882194 5791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ec93e8e482459391c9890a727d49df, State: Initialized, Role: FOLLOWER
I20260620 14:20:45.882613 5791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ec93e8e482459391c9890a727d49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } }
I20260620 14:20:45.882762 5791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:45.882848 5791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:45.882965 5791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:45.883867 5791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ec93e8e482459391c9890a727d49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } }
I20260620 14:20:45.884197 5791 leader_election.cc:302] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42ec93e8e482459391c9890a727d49df; no voters:
I20260620 14:20:45.884512 5791 leader_election.cc:288] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:45.884739 5796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:45.885020 5796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [term 1 LEADER]: Becoming Leader. State: Replica: 42ec93e8e482459391c9890a727d49df, State: Running, Role: LEADER
I20260620 14:20:45.885301 5796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ec93e8e482459391c9890a727d49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } }
I20260620 14:20:45.885751 5791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:45.886852 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42ec93e8e482459391c9890a727d49df. Latest consensus state: current_term: 1 leader_uuid: "42ec93e8e482459391c9890a727d49df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ec93e8e482459391c9890a727d49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } } }
I20260620 14:20:45.886996 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:45.887274 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42ec93e8e482459391c9890a727d49df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ec93e8e482459391c9890a727d49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } } }
I20260620 14:20:45.887365 5805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:45.887391 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:45.890340 5805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:45.896250 5805 catalog_manager.cc:1347] Generated new cluster ID: e11e84ab2b1e449bbe16adcc48a6e1ef
I20260620 14:20:45.896329 5805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:45.908986 5805 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:45.909870 5805 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:45.914667 5805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42ec93e8e482459391c9890a727d49df: Generated new TSK 0
I20260620 14:20:45.915359 5805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:20:45.936832 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:45.931760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.0.98.62:43303)
0620 14:20:45.931898 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:45.931903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:45.932074 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:20:45.932203 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:45.932206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:45.932265 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:45.932384 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:20:45.932586 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.933410 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.934257 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.934611 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.934714 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:45.934968 (+ 254us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:20:45.936454 (+ 1486us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:20:45.936682 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 14:20:45.936905 5794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:20:45.938556 5820 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:45.931631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43303 (local address 127.0.0.1:49506)
0620 14:20:45.931990 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:45.932001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:45.932094 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:45.932386 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:45.932389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:45.932395 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:20:45.932485 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:45.932490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.933538 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.933540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:45.934152 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:45.934159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:45.934706 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:45.934725 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:45.934782 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:45.934803 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:20:45.936614 (+ 1811us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:20:45.938400 (+ 1786us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":1544,"thread_start_us":129,"threads_started":1}
I20260620 14:20:45.939172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 5724
E20260620 14:20:45.947127 5821 webserver.cc:533] Webserver: sslize failed:
E20260620 14:20:45.947198 5821 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:20:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (308 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:45Z Disabled control of system clock
I20260620 14:20:45.986775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40455
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40455
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/testchainca.cert with env {}
W20260620 14:20:46.093854 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:46.094117 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:46.094156 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:46.098057 5825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:46.098105 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:46.098120 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:46.098137 5825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:46.098152 5825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:20:46.098167 5825 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:20:46.098181 5825 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:20:46.103309 5825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:46.104312 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:46.105237 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:46.113447 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:46.113409 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:46.113448 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:46.113795 5825 server_base.cc:1034] running on GCE node
I20260620 14:20:46.114399 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:46.115897 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:46.117055 5825 hybrid_clock.cc:648] HybridClock initialized: now 1781965246117025 us; error 44 us; skew 500 ppm
I20260620 14:20:46.118911 5825 webserver.cc:460] Webserver started at http://127.0.98.62:34907/ using document root <none> and password file <none>
I20260620 14:20:46.119479 5825 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:46.119560 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:46.119848 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:46.121836 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "ae0b521fbf564582a62a56a4d54d4f68"
format_stamp: "Formatted at 2026-06-20 14:20:46 on dist-test-slave-87nc"
server_key: "99f22517150d6c718462ce959e29f9f8"
server_key_iv: "1db79682a1b169b8a05af06bead77d6f"
server_key_version: "encryptionkey@0"
I20260620 14:20:46.122392 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae0b521fbf564582a62a56a4d54d4f68"
format_stamp: "Formatted at 2026-06-20 14:20:46 on dist-test-slave-87nc"
server_key: "99f22517150d6c718462ce959e29f9f8"
server_key_iv: "1db79682a1b169b8a05af06bead77d6f"
server_key_version: "encryptionkey@0"
I20260620 14:20:46.125866 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:20:46.128151 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:46.129343 5825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:46.129483 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "ae0b521fbf564582a62a56a4d54d4f68"
format_stamp: "Formatted at 2026-06-20 14:20:46 on dist-test-slave-87nc"
server_key: "99f22517150d6c718462ce959e29f9f8"
server_key_iv: "1db79682a1b169b8a05af06bead77d6f"
server_key_version: "encryptionkey@0"
I20260620 14:20:46.129601 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:46.141191 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:46.141800 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:46.141989 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:46.148458 5825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40455
I20260620 14:20:46.148468 5891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40455 every 8 connection(s)
I20260620 14:20:46.149371 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:46.152352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 5825
I20260620 14:20:46.152480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:46.152753 5892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:46.152799 392 external_mini_cluster.cc:1428] Setting key b3d80f3d3f27465bae48e4bfb403d3d2
I20260620 14:20:46.159085 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68: Bootstrap starting.
I20260620 14:20:46.161485 5892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:46.161491 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:46.154054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.0.98.62:40455)
0620 14:20:46.154624 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:46.154640 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:46.154675 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:20:46.155286 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:46.155301 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:46.155673 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:46.156069 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:46.156473 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:46.157420 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:46.158286 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:46.158744 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:46.158921 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:46.159821 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:46.159827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:46.159848 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:20:46.160219 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:46.160272 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:20:46.160291 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:46.160404 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:46.160574 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:20:46.160818 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260620 14:20:46.162228 5892 log.cc:824] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:46.164403 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:20:46.164506 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68: No bootstrap required, opened a new log
I20260620 14:20:46.168490 5892 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0b521fbf564582a62a56a4d54d4f68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40455 } }
I20260620 14:20:46.168730 5892 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:46.168798 5892 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b521fbf564582a62a56a4d54d4f68, State: Initialized, Role: FOLLOWER
I20260620 14:20:46.169194 5892 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0b521fbf564582a62a56a4d54d4f68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40455 } }
I20260620 14:20:46.169342 5892 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:46.169438 5892 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:46.169538 5892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:46.170425 5892 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0b521fbf564582a62a56a4d54d4f68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40455 } }
I20260620 14:20:46.170763 5892 leader_election.cc:302] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae0b521fbf564582a62a56a4d54d4f68; no voters:
I20260620 14:20:46.171098 5892 leader_election.cc:288] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:46.171227 5897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:46.171505 5897 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b521fbf564582a62a56a4d54d4f68, State: Running, Role: LEADER
I20260620 14:20:46.171887 5897 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0b521fbf564582a62a56a4d54d4f68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40455 } }
I20260620 14:20:46.172519 5892 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:46.173676 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae0b521fbf564582a62a56a4d54d4f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0b521fbf564582a62a56a4d54d4f68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40455 } } }
I20260620 14:20:46.173825 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:46.174108 5899 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae0b521fbf564582a62a56a4d54d4f68. Latest consensus state: current_term: 1 leader_uuid: "ae0b521fbf564582a62a56a4d54d4f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0b521fbf564582a62a56a4d54d4f68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40455 } } }
I20260620 14:20:46.174203 5906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:46.174211 5899 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:46.177280 5906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:46.183077 5906 catalog_manager.cc:1347] Generated new cluster ID: 0f2fdc781de54b61aa082492e48fe706
I20260620 14:20:46.183162 5906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:46.191869 5906 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:46.192759 5906 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:46.197458 5906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae0b521fbf564582a62a56a4d54d4f68: Generated new TSK 0
I20260620 14:20:46.198158 5906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:20:46.258016 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:46.211835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.0.98.62:40455)
0620 14:20:46.211973 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:46.211977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:46.212200 (+ 223us) server_negotiation.cc:407] Connection header received
0620 14:20:46.212346 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:46.212349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:46.212399 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:46.212483 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:20:46.212740 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:46.213248 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:46.214506 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:46.214700 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:46.214775 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:46.215086 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:20:46.257636 (+ 42550us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:20:46.257901 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39472: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 14:20:46.258122 5895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39472: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:20:46.258289 5921 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:46.211789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40455 (local address 127.0.0.1:39472)
0620 14:20:46.212107 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:46.212122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:46.212235 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:46.212509 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:46.212512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:46.212521 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:20:46.212626 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:46.212633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:46.213349 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:46.213352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:46.214397 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:46.214408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:46.214800 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:46.214820 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:46.214909 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:46.214943 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:20:46.257876 (+ 42933us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:20:46.258096 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40455: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":126,"threads_started":1}
I20260620 14:20:46.258935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 5825
2026-06-20T14:20:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (298 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:46 dist-test-slave-87nc krb5kdc[5933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:46 dist-test-slave-87nc krb5kdc[5933](info): set up 2 sockets
Jun 20 14:20:46 dist-test-slave-87nc krb5kdc[5933](info): commencing operation
krb5kdc: starting...
W20260620 14:20:47.536329 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:47 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:47.655054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40805
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:47.762828 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:47.763152 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:47.763224 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:47.767278 5946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:47.767391 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:47.767429 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:47.767493 5946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:47.767531 5946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:47.773590 5946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:47.774777 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:47.775864 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:47.784615 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:47.784673 5946 server_base.cc:1034] running on GCE node
W20260620 14:20:47.784622 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:47.784622 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:47.785454 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:47.786589 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:47.787861 5946 hybrid_clock.cc:648] HybridClock initialized: now 1781965247787844 us; error 31 us; skew 500 ppm
Jun 20 14:20:47 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:47.794363 5946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:47.795461 5946 webserver.cc:460] Webserver started at http://127.0.98.62:41833/ using document root <none> and password file <none>
I20260620 14:20:47.796128 5946 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:47.796233 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:47.796492 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:47.798655 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "967fabb423284edf8d63c53a1c5a2cfe"
format_stamp: "Formatted at 2026-06-20 14:20:47 on dist-test-slave-87nc"
server_key: "bc94db99e464b723945dc5cc54561fe0"
server_key_iv: "77a4c5311695e45b9b2d1386bb6852e2"
server_key_version: "encryptionkey@0"
I20260620 14:20:47.799294 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "967fabb423284edf8d63c53a1c5a2cfe"
format_stamp: "Formatted at 2026-06-20 14:20:47 on dist-test-slave-87nc"
server_key: "bc94db99e464b723945dc5cc54561fe0"
server_key_iv: "77a4c5311695e45b9b2d1386bb6852e2"
server_key_version: "encryptionkey@0"
I20260620 14:20:47.803197 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 14:20:47.805886 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:47.807184 5946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:20:47.807340 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "967fabb423284edf8d63c53a1c5a2cfe"
format_stamp: "Formatted at 2026-06-20 14:20:47 on dist-test-slave-87nc"
server_key: "bc94db99e464b723945dc5cc54561fe0"
server_key_iv: "77a4c5311695e45b9b2d1386bb6852e2"
server_key_version: "encryptionkey@0"
I20260620 14:20:47.807478 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:47.823086 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:47.826529 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:47.826742 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:47.833730 5946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40805
I20260620 14:20:47.834031 6013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40805 every 8 connection(s)
I20260620 14:20:47.834726 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:47.837618 6014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:47.840557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 5946
I20260620 14:20:47.840695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:47.840999 392 external_mini_cluster.cc:1428] Setting key 96bef1b3ce4e9d09be77efe67e7c35ca
I20260620 14:20:47.843755 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe: Bootstrap starting.
I20260620 14:20:47.846473 6014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:47.847360 6014 log.cc:824] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:47.849470 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe: No bootstrap required, opened a new log
Jun 20 14:20:47 dist-test-slave-87nc krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:47.853739 6014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967fabb423284edf8d63c53a1c5a2cfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40805 } }
I20260620 14:20:47.854027 6014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:47.854079 6014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967fabb423284edf8d63c53a1c5a2cfe, State: Initialized, Role: FOLLOWER
I20260620 14:20:47.854573 6014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967fabb423284edf8d63c53a1c5a2cfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40805 } }
I20260620 14:20:47.854763 6014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:47.854869 6014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:47.854974 6014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:47.856107 6014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967fabb423284edf8d63c53a1c5a2cfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40805 } }
I20260620 14:20:47.856271 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:47.842387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.0.98.62:40805)
0620 14:20:47.842798 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:47.842814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:47.842849 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:20:47.843453 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:47.843470 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:47.843861 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:47.844196 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:47.845050 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:47.846010 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:47.847196 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:47.847805 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:47.847943 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:47.850519 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:47.850525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:47.850541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:47.850570 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:47.853092 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:47.853512 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:47.853517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:47.853523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:47.853659 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:47.853910 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:47.853913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:47.853915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:47.854418 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:47.854605 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:47.854949 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 14:20:47.855272 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
I20260620 14:20:47.856451 6014 leader_election.cc:302] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 967fabb423284edf8d63c53a1c5a2cfe; no voters:
I20260620 14:20:47.856769 6014 leader_election.cc:288] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:47.856889 6019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:47.857133 6019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [term 1 LEADER]: Becoming Leader. State: Replica: 967fabb423284edf8d63c53a1c5a2cfe, State: Running, Role: LEADER
I20260620 14:20:47.857969 6014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:47.858511 6019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967fabb423284edf8d63c53a1c5a2cfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40805 } }
I20260620 14:20:47.860369 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 967fabb423284edf8d63c53a1c5a2cfe. Latest consensus state: current_term: 1 leader_uuid: "967fabb423284edf8d63c53a1c5a2cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967fabb423284edf8d63c53a1c5a2cfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40805 } } }
I20260620 14:20:47.860800 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:47.860353 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "967fabb423284edf8d63c53a1c5a2cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967fabb423284edf8d63c53a1c5a2cfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40805 } } }
I20260620 14:20:47.861038 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe [sys.catalog]: This master's current role is: LEADER
W20260620 14:20:47.862391 6033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:20:47.862454 6033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:20:47.862560 6034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:47.863401 6034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:47.868844 6034 catalog_manager.cc:1347] Generated new cluster ID: 01be29dc988e48d7b0ae75c77348305c
I20260620 14:20:47.868942 6034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:47.884195 6034 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:47.885473 6034 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:47.894330 6034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 967fabb423284edf8d63c53a1c5a2cfe: Generated new TSK 0
I20260620 14:20:47.895071 6034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:20:47.934701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40805
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:48.051028 6040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:48.051283 6040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:48.051326 6040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:48.055373 6040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:48.055433 6040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:48.055490 6040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:48.060943 6040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:48.061997 6040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:48.062978 6040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:48.072593 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:48.072552 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:48.072563 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:48.072911 6040 server_base.cc:1034] running on GCE node
I20260620 14:20:48.073278 6040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:48.073904 6040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:48.075085 6040 hybrid_clock.cc:648] HybridClock initialized: now 1781965248075073 us; error 33 us; skew 500 ppm
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.077800 6040 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:48.078836 6040 webserver.cc:460] Webserver started at http://127.0.98.1:41163/ using document root <none> and password file <none>
I20260620 14:20:48.079385 6040 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:48.079473 6040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:48.079682 6040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:48.081786 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "e70bd1057e4a4973a932ed3969bd5e89"
format_stamp: "Formatted at 2026-06-20 14:20:48 on dist-test-slave-87nc"
server_key: "b18c4286e9b19d55d13b47cf0b003b9c"
server_key_iv: "b5b1db8d5d1febcf91941ca3409af1e1"
server_key_version: "encryptionkey@0"
I20260620 14:20:48.082340 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e70bd1057e4a4973a932ed3969bd5e89"
format_stamp: "Formatted at 2026-06-20 14:20:48 on dist-test-slave-87nc"
server_key: "b18c4286e9b19d55d13b47cf0b003b9c"
server_key_iv: "b5b1db8d5d1febcf91941ca3409af1e1"
server_key_version: "encryptionkey@0"
I20260620 14:20:48.085917 6040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:20:48.088281 6055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:48.089627 6040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:48.089821 6040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "e70bd1057e4a4973a932ed3969bd5e89"
format_stamp: "Formatted at 2026-06-20 14:20:48 on dist-test-slave-87nc"
server_key: "b18c4286e9b19d55d13b47cf0b003b9c"
server_key_iv: "b5b1db8d5d1febcf91941ca3409af1e1"
server_key_version: "encryptionkey@0"
I20260620 14:20:48.089944 6040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:48.120057 6040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:48.123122 6040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:48.123332 6040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:48.123992 6040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:48.124809 6040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:48.124879 6040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:48.124951 6040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:48.124995 6040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:48.133687 6040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33719
I20260620 14:20:48.133713 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33719 every 8 connection(s)
I20260620 14:20:48.134562 6040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:48.141880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 6040
I20260620 14:20:48.142021 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:48.142356 392 external_mini_cluster.cc:1428] Setting key 9ba668acc39bb77ffb116de5212a11b6
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:48.148684 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:48.136387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49265 (local address 127.0.98.62:40805)
0620 14:20:48.136548 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:48.136552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:48.137443 (+ 891us) server_negotiation.cc:407] Connection header received
0620 14:20:48.138721 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:48.138723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:48.138767 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:48.138844 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:48.139958 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.140460 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.141513 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.141718 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.141806 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:48.144757 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:48.144764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:48.144768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:48.144804 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:48.146393 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.147053 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.147058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.147059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.147149 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.147521 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.147526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.147528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.147723 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:48.147872 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:48.148372 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 14:20:48.148517 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:20:48.149561 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:48.136826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40805 (local address 127.0.98.1:49265)
0620 14:20:48.137274 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:48.137328 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:48.138533 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:48.138999 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:48.139005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:48.139348 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:48.139839 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:48.139853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.140560 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.140562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:48.141418 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:48.141426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.141882 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.141887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:48.142105 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:48.142753 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:48.142773 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:48.144594 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:48.146511 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:48.146519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:48.146537 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:48.146917 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:48.147254 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:48.147257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:48.147259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:48.147417 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:48.147871 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:48.147879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:48.148135 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 14:20:48.148333 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 14:20:48.148675 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260620 14:20:48.150768 6169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40805
I20260620 14:20:48.151026 6169 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:48.151541 6169 heartbeater.cc:511] Master 127.0.98.62:40805 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:20:48.153134 5978 ts_manager.cc:194] Registered new tserver with Master: e70bd1057e4a4973a932ed3969bd5e89 (127.0.98.1:33719)
I20260620 14:20:48.154341 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:20:48.169124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40805
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:48.281531 6174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:48.281832 6174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:48.281898 6174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:48.285857 6174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:48.285921 6174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:48.285980 6174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:48.291834 6174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:48.292879 6174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:48.293959 6174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:48.303694 6179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:48.303690 6182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:48.303681 6180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:48.304013 6174 server_base.cc:1034] running on GCE node
I20260620 14:20:48.304436 6174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:48.305075 6174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:48.306275 6174 hybrid_clock.cc:648] HybridClock initialized: now 1781965248306256 us; error 35 us; skew 500 ppm
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.309105 6174 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:48.310014 6174 webserver.cc:460] Webserver started at http://127.0.98.2:41581/ using document root <none> and password file <none>
I20260620 14:20:48.310585 6174 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:48.310722 6174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:48.310974 6174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:48.313601 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "fc13286c0e094aaaade98ba0d97eca6e"
format_stamp: "Formatted at 2026-06-20 14:20:48 on dist-test-slave-87nc"
server_key: "dfaa992b76aadded42465c92de36078b"
server_key_iv: "7bc80c0e2fde0494078a27ca56c13b83"
server_key_version: "encryptionkey@0"
I20260620 14:20:48.314221 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fc13286c0e094aaaade98ba0d97eca6e"
format_stamp: "Formatted at 2026-06-20 14:20:48 on dist-test-slave-87nc"
server_key: "dfaa992b76aadded42465c92de36078b"
server_key_iv: "7bc80c0e2fde0494078a27ca56c13b83"
server_key_version: "encryptionkey@0"
I20260620 14:20:48.318058 6174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:20:48.320762 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:48.321928 6174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:20:48.322062 6174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "fc13286c0e094aaaade98ba0d97eca6e"
format_stamp: "Formatted at 2026-06-20 14:20:48 on dist-test-slave-87nc"
server_key: "dfaa992b76aadded42465c92de36078b"
server_key_iv: "7bc80c0e2fde0494078a27ca56c13b83"
server_key_version: "encryptionkey@0"
I20260620 14:20:48.322170 6174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:48.336150 6174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:48.339123 6174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:48.339349 6174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:48.340011 6174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:48.340839 6174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:48.340909 6174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:48.340974 6174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:48.341017 6174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:48.349706 6174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41679
I20260620 14:20:48.349720 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41679 every 8 connection(s)
I20260620 14:20:48.350566 6174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:48.355713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 6174
I20260620 14:20:48.355896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:48.356156 392 external_mini_cluster.cc:1428] Setting key f580b3015c80f7c7686c76b8f41c2da1
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:48.365759 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:48.352319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41439 (local address 127.0.98.62:40805)
0620 14:20:48.352453 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:48.352458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:48.353212 (+ 754us) server_negotiation.cc:407] Connection header received
0620 14:20:48.354567 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:48.354570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:48.354628 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:48.354739 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:48.356152 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.356875 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.358529 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.358747 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.358832 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:48.361783 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:48.361790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:48.361796 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:48.361829 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:48.363545 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.364108 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.364112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.364114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.364210 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.364539 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.364544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.364546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.364803 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:48.364921 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:48.365393 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 14:20:48.365530 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:20:48.366533 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:48.352642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40805 (local address 127.0.98.2:41439)
0620 14:20:48.353065 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:48.353107 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:48.354351 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:48.354905 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:48.354913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:48.355256 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:48.356011 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:48.356028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.357005 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.357008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:48.358378 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:48.358392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.358898 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.358904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:48.359188 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:48.359860 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:48.359886 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:48.361591 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:48.363663 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:48.363667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:48.363679 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:48.364007 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:48.364304 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:48.364306 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:48.364308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:48.364435 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:48.364908 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:48.364912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:48.365139 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 14:20:48.365283 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:20:48.365530 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260620 14:20:48.367726 6303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40805
I20260620 14:20:48.368029 6303 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:48.368582 6303 heartbeater.cc:511] Master 127.0.98.62:40805 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:20:48.369720 5978 ts_manager.cc:194] Registered new tserver with Master: fc13286c0e094aaaade98ba0d97eca6e (127.0.98.2:41679)
I20260620 14:20:48.370550 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:20:48.384516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40805
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:48.497661 6308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:48.497938 6308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:48.498023 6308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:48.502238 6308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:48.502297 6308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:48.502390 6308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:48.507598 6308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:48.508706 6308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:48.509744 6308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:48.519639 6313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:48.519696 6308 server_base.cc:1034] running on GCE node
W20260620 14:20:48.519634 6316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:48.519948 6314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:48.520406 6308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:48.521008 6308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:48.522202 6308 hybrid_clock.cc:648] HybridClock initialized: now 1781965248522188 us; error 30 us; skew 500 ppm
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.524966 6308 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:48.525842 6308 webserver.cc:460] Webserver started at http://127.0.98.3:44795/ using document root <none> and password file <none>
I20260620 14:20:48.526371 6308 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:48.526427 6308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:48.526587 6308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:48.528614 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "9dbbedf230b24028a075b17fd041a4fa"
format_stamp: "Formatted at 2026-06-20 14:20:48 on dist-test-slave-87nc"
server_key: "69659166082a9da27939b856b2dfd83d"
server_key_iv: "37b8104de8147c73986f398cc13eb126"
server_key_version: "encryptionkey@0"
I20260620 14:20:48.529119 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9dbbedf230b24028a075b17fd041a4fa"
format_stamp: "Formatted at 2026-06-20 14:20:48 on dist-test-slave-87nc"
server_key: "69659166082a9da27939b856b2dfd83d"
server_key_iv: "37b8104de8147c73986f398cc13eb126"
server_key_version: "encryptionkey@0"
I20260620 14:20:48.532789 6308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:20:48.535091 6323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:48.536332 6308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:48.536448 6308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "9dbbedf230b24028a075b17fd041a4fa"
format_stamp: "Formatted at 2026-06-20 14:20:48 on dist-test-slave-87nc"
server_key: "69659166082a9da27939b856b2dfd83d"
server_key_iv: "37b8104de8147c73986f398cc13eb126"
server_key_version: "encryptionkey@0"
I20260620 14:20:48.536533 6308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:48.553231 6308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:48.556068 6308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:48.556221 6308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:48.556751 6308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:48.557565 6308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:48.557612 6308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:48.557654 6308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:48.557674 6308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:48.566684 6308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46457
I20260620 14:20:48.566702 6436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46457 every 8 connection(s)
I20260620 14:20:48.567590 6308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:48.570991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 6308
I20260620 14:20:48.571094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:48.571388 392 external_mini_cluster.cc:1428] Setting key 434fbb4c2200b7885313927c98f5f217
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:48.580689 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:48.569364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49329 (local address 127.0.98.62:40805)
0620 14:20:48.569508 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:48.569511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:48.570315 (+ 804us) server_negotiation.cc:407] Connection header received
0620 14:20:48.571797 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:48.571800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:48.571857 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:48.571946 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:48.573079 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.573617 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.574771 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.575000 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.575113 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:48.577508 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:48.577512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:48.577514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:48.577547 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:48.578759 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.579318 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.579321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.579322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.579403 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.579709 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.579712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.579713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.579937 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:48.580015 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:48.580474 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 14:20:48.580571 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:20:48.581568 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:48.569738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40805 (local address 127.0.98.3:49329)
0620 14:20:48.570166 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:48.570208 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:48.571572 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:48.572132 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:48.572138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:48.572478 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:48.572942 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:48.572954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.573744 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.573752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:48.574645 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:48.574654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.575146 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.575150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:48.575323 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:48.575956 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:48.575974 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:48.577346 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:48.578881 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:48.578886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:48.578916 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:48.579215 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:48.579491 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:48.579494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:48.579495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:48.579627 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:48.580026 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:48.580031 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:48.580267 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 14:20:48.580434 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:20:48.580689 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260620 14:20:48.582737 6437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40805
I20260620 14:20:48.582983 6437 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:48.583472 6437 heartbeater.cc:511] Master 127.0.98.62:40805 requested a full tablet report, sending...
I20260620 14:20:48.584694 5978 ts_manager.cc:194] Registered new tserver with Master: 9dbbedf230b24028a075b17fd041a4fa (127.0.98.3:46457)
I20260620 14:20:48.585332 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49329
I20260620 14:20:48.596815 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/test-admin.keytab.
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.611434 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:20:48.614315 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:48.612974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.0.98.1:33719)
0620 14:20:48.613283 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:48.613296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:48.613319 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:20:48.613366 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:48.613371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:48.613573 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:48.613693 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:48.614111 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48172: BlockingRecv error: recv got EOF from 127.0.0.1:48172 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":127,"threads_started":1}
W20260620 14:20:48.614564 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5cc)
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.615600 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.616590 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:48.626770 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:48.617377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.0.98.1:33719)
0620 14:20:48.617505 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:48.617509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:48.617622 (+ 113us) server_negotiation.cc:407] Connection header received
0620 14:20:48.617756 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:48.617759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:48.617806 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:48.617890 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:48.618424 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.619246 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.620252 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.620627 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.620768 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:48.622932 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:48.622940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:48.622953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:48.622974 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:48.624479 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.625037 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.625044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.625046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.625144 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.625480 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.625485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.625486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.625814 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:48.626033 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:48.626300 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:20:48.626516 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 14:20:48.631014 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:48.629685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.0.98.1:33719)
0620 14:20:48.629804 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:48.629808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:48.630069 (+ 261us) server_negotiation.cc:407] Connection header received
0620 14:20:48.630178 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:48.630181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:48.630284 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:48.630367 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:48.630873 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48182: BlockingRecv error: recv got EOF from 127.0.0.1:48182 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:20:48.631079 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.632247 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.633172 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:48.641795 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:48.633996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.0.98.1:33719)
0620 14:20:48.634117 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:48.634121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:48.634222 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:20:48.634364 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:48.634367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:48.634407 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:48.634497 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:48.634980 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.635495 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.636368 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.636602 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.636705 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:48.638758 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:48.638762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:48.638765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:48.638797 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:48.640150 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.640566 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.640570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.640572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.640654 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.640927 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.640930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.640932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.641138 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:48.641279 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:48.641518 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:20:48.641629 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 14:20:48.644287 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:48.643411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.0.98.1:33719)
0620 14:20:48.643534 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:48.643537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:48.643579 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:20:48.643667 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:48.643669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:48.643701 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:48.643826 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:48.644202 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48194: BlockingRecv error: recv got EOF from 127.0.0.1:48194 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 14:20:48.644410 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965214882667-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.645432 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:48.646291 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:20:48.654567 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:48.647065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.0.98.1:33719)
0620 14:20:48.647183 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:48.647187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:48.647303 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:20:48.647456 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:48.647459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:48.647501 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:48.647587 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:48.648134 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.648619 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.649390 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:48.649604 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:48.649708 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:48.651829 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:48.651833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:48.651836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:48.651867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:48.653017 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.653437 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.653441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.653443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.653518 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:48.653785 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:48.653790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:48.653792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:48.653956 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:48.654089 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:48.654312 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:20:48.654414 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:20:48.655489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 6040
I20260620 14:20:48.661319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 6174
I20260620 14:20:48.666473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 6308
I20260620 14:20:48.673029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 5946
2026-06-20T14:20:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2175 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[6457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[6457](info): set up 2 sockets
Jun 20 14:20:48 dist-test-slave-87nc krb5kdc[6457](info): commencing operation
krb5kdc: starting...
W20260620 14:20:49.700237 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:49 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:20:49.818632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:49.926117 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:49.926422 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:49.926497 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:49.930508 6470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:49.930608 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:49.930644 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:49.930683 6470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:49.930716 6470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:49.935948 6470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:49.937078 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:49.938165 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:49.946725 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:49.946748 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:49.946744 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:49.946961 6470 server_base.cc:1034] running on GCE node
I20260620 14:20:49.947693 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:49.949296 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:49.950488 6470 hybrid_clock.cc:648] HybridClock initialized: now 1781965249950472 us; error 36 us; skew 500 ppm
Jun 20 14:20:49 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:49.953259 6470 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:20:49.954214 6470 webserver.cc:460] Webserver started at http://127.0.98.62:40539/ using document root <none> and password file <none>
I20260620 14:20:49.954722 6470 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:49.954780 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:49.954950 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:49.957010 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "71a466f760d746b2b42fe1b85085da5f"
format_stamp: "Formatted at 2026-06-20 14:20:49 on dist-test-slave-87nc"
server_key: "f6f5c884da05493e90505196c95df49c"
server_key_iv: "463a7e00a0888a8106c8abbea64f0621"
server_key_version: "encryptionkey@0"
I20260620 14:20:49.957536 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "71a466f760d746b2b42fe1b85085da5f"
format_stamp: "Formatted at 2026-06-20 14:20:49 on dist-test-slave-87nc"
server_key: "f6f5c884da05493e90505196c95df49c"
server_key_iv: "463a7e00a0888a8106c8abbea64f0621"
server_key_version: "encryptionkey@0"
I20260620 14:20:49.961172 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:20:49.963480 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:49.964771 6470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:49.964915 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "71a466f760d746b2b42fe1b85085da5f"
format_stamp: "Formatted at 2026-06-20 14:20:49 on dist-test-slave-87nc"
server_key: "f6f5c884da05493e90505196c95df49c"
server_key_iv: "463a7e00a0888a8106c8abbea64f0621"
server_key_version: "encryptionkey@0"
I20260620 14:20:49.965021 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:49.974989 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:49.977780 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:49.977995 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:49.985044 6470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40575
I20260620 14:20:49.985323 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40575 every 8 connection(s)
I20260620 14:20:49.986063 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:49.988811 6538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:49.994136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 6470
I20260620 14:20:49.994272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:49.994577 392 external_mini_cluster.cc:1428] Setting key dcdfe2aef02f6314ba7a7bbce377deb6
I20260620 14:20:49.995271 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: Bootstrap starting.
I20260620 14:20:49.997634 6538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:49.998356 6538 log.cc:824] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:50.000242 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: No bootstrap required, opened a new log
Jun 20 14:20:49 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:50.004303 6538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } }
I20260620 14:20:50.004606 6538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:50.004671 6538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a466f760d746b2b42fe1b85085da5f, State: Initialized, Role: FOLLOWER
I20260620 14:20:50.005149 6538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } }
I20260620 14:20:50.005311 6538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:50.005379 6538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:50.005476 6538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:50.006349 6538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } }
I20260620 14:20:50.006698 6538 leader_election.cc:302] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71a466f760d746b2b42fe1b85085da5f; no voters:
I20260620 14:20:50.007009 6538 leader_election.cc:288] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:50.007117 6543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:50.007339 6543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 1 LEADER]: Becoming Leader. State: Replica: 71a466f760d746b2b42fe1b85085da5f, State: Running, Role: LEADER
I20260620 14:20:50.007725 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } }
I20260620 14:20:50.008369 6538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:50.009078 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:49.995969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.0.98.62:40575)
0620 14:20:49.996391 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:49.996406 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:49.996441 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:20:49.997053 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:49.997070 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:49.997407 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:49.997750 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:49.998545 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:49.999513 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.000484 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.001154 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.001319 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.003662 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:50.003671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:50.003690 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:50.003719 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:50.005781 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.006232 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.006241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.006251 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.006363 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.006660 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.006665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.006667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.007191 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:50.007416 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:50.007658 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:20:50.008013 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":129,"threads_started":1}
I20260620 14:20:50.009167 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } }
I20260620 14:20:50.009284 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:50.010005 6544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71a466f760d746b2b42fe1b85085da5f. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } }
I20260620 14:20:50.010154 6544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:50.011418 6548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:50.014473 6548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:50.019867 6548 catalog_manager.cc:1347] Generated new cluster ID: ccd44411486a4584808cf9a7c46c2a6e
I20260620 14:20:50.019951 6548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:50.031299 6548 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:50.032374 6548 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:50.046182 6548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: Generated new TSK 0
I20260620 14:20:50.046973 6548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:20:50.078688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40575
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:50.189273 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:50.189519 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:50.189563 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:50.193549 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:50.193607 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:50.193706 6564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:50.199088 6564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:50.200155 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:50.201087 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:20:50.210443 6564 server_base.cc:1034] running on GCE node
W20260620 14:20:50.210407 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:50.210369 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:50.210364 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:50.211027 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:50.211609 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:50.212798 6564 hybrid_clock.cc:648] HybridClock initialized: now 1781965250212783 us; error 31 us; skew 500 ppm
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:50.215472 6564 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:20:50.216346 6564 webserver.cc:460] Webserver started at http://127.0.98.1:35651/ using document root <none> and password file <none>
I20260620 14:20:50.216862 6564 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:50.216919 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:50.217077 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:50.219019 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "e15ef7c8dfbf4028bd99881804e1bc13"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "8f086aaf413f58514500b5390cb90f1c"
server_key_iv: "99a7a0ee05fc7f03165bbdc17b62edf3"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.219527 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e15ef7c8dfbf4028bd99881804e1bc13"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "8f086aaf413f58514500b5390cb90f1c"
server_key_iv: "99a7a0ee05fc7f03165bbdc17b62edf3"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.223094 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:20:50.225504 6579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.226717 6564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:20:50.226811 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "e15ef7c8dfbf4028bd99881804e1bc13"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "8f086aaf413f58514500b5390cb90f1c"
server_key_iv: "99a7a0ee05fc7f03165bbdc17b62edf3"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.226903 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:50.245947 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:50.249063 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:50.249223 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:50.249783 6564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:50.250561 6564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:50.250608 6564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.250648 6564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:50.250662 6564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.259603 6564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32929
I20260620 14:20:50.259622 6692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32929 every 8 connection(s)
I20260620 14:20:50.260571 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:50.264940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 6564
I20260620 14:20:50.265072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:50.265362 392 external_mini_cluster.cc:1428] Setting key a52240856b15727b6f2a9f1326932536
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:20:50.275822 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.262368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35715 (local address 127.0.98.62:40575)
0620 14:20:50.262500 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:50.262503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:50.263362 (+ 859us) server_negotiation.cc:407] Connection header received
0620 14:20:50.264713 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:50.264718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:50.264778 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:50.264880 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:50.266193 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.266938 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.268703 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.269133 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.269270 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.272120 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:50.272130 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:50.272134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:50.272157 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:50.273654 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.274195 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.274204 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.274205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.274281 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.274636 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.274639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.274641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.274829 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:50.274947 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:50.275482 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 14:20:50.275617 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:20:50.276777 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.262802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:35715)
0620 14:20:50.263214 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:50.263255 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:50.264502 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:50.265030 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:50.265036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:50.265502 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:50.266045 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.266071 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.267059 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.267063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.268585 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.268594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.269324 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.269328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.269483 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.270108 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:50.270133 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:50.271955 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:50.273789 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.273794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.273811 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.274083 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.274376 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.274389 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.274392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.274534 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.274946 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:50.274955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:50.275231 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 14:20:50.275431 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 14:20:50.275794 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 14:20:50.278110 6693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40575
I20260620 14:20:50.278381 6693 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:50.278877 6693 heartbeater.cc:511] Master 127.0.98.62:40575 requested a full tablet report, sending...
I20260620 14:20:50.280586 6502 ts_manager.cc:194] Registered new tserver with Master: e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929)
I20260620 14:20:50.281737 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:20:50.292304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40575
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:50.398994 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:50.399238 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:50.399281 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:50.403102 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:50.403152 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:50.403208 6698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:50.408304 6698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:50.409273 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:50.410236 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:50.419230 6704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:50.419232 6706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:50.419318 6698 server_base.cc:1034] running on GCE node
W20260620 14:20:50.419236 6703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:50.419854 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:50.420454 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:50.421634 6698 hybrid_clock.cc:648] HybridClock initialized: now 1781965250421621 us; error 24 us; skew 500 ppm
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:50.424352 6698 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:20:50.425212 6698 webserver.cc:460] Webserver started at http://127.0.98.2:38995/ using document root <none> and password file <none>
I20260620 14:20:50.425699 6698 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:50.425769 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:50.425932 6698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:50.428004 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "96f5529d6fb143cdae32db52d94cc28d"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "85a51f50cd18904e2b958d0b61e490fb"
server_key_iv: "b9265060874b117f8e9622c5976ed21d"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.428514 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "96f5529d6fb143cdae32db52d94cc28d"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "85a51f50cd18904e2b958d0b61e490fb"
server_key_iv: "b9265060874b117f8e9622c5976ed21d"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.432145 6698 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:20:50.434352 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.435465 6698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:50.435554 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "96f5529d6fb143cdae32db52d94cc28d"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "85a51f50cd18904e2b958d0b61e490fb"
server_key_iv: "b9265060874b117f8e9622c5976ed21d"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.435628 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:50.453423 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:50.456666 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:50.456817 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:50.457337 6698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:50.458104 6698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:50.458150 6698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.458189 6698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:50.458209 6698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.467062 6698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35525
I20260620 14:20:50.467082 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35525 every 8 connection(s)
I20260620 14:20:50.468108 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:50.468391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 6698
I20260620 14:20:50.468526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:50.468904 392 external_mini_cluster.cc:1428] Setting key af8f357ae732ba6401bfa7214bcebad1
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:50.486670 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.470159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35615 (local address 127.0.98.62:40575)
0620 14:20:50.470307 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:50.470312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:50.471442 (+ 1130us) server_negotiation.cc:407] Connection header received
0620 14:20:50.473315 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:50.473318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:50.473373 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:50.473449 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:50.475180 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.475708 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.477566 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.477792 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.477874 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.482846 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:50.482849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:50.482852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:50.482874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:50.484383 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.485002 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.485005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.485006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.485071 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.485489 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.485494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.485496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.485690 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:50.485765 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:50.486419 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 14:20:50.486532 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:20:50.488036 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.470748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:35615)
0620 14:20:50.471274 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:50.471318 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:50.473052 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:50.473665 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:50.473677 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:50.474261 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:50.475015 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.475039 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.475884 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.475891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.477444 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.477456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.477957 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.477961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.478166 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.478767 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:50.478793 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:50.482670 (+ 3877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:50.484499 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.484508 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.484526 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.484904 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.485160 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.485166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.485170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.485367 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.485800 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:50.485808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:50.486151 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 14:20:50.486389 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 14:20:50.486806 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":164,"threads_started":1}
I20260620 14:20:50.489312 6827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40575
I20260620 14:20:50.489553 6827 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:50.490020 6827 heartbeater.cc:511] Master 127.0.98.62:40575 requested a full tablet report, sending...
I20260620 14:20:50.491027 6502 ts_manager.cc:194] Registered new tserver with Master: 96f5529d6fb143cdae32db52d94cc28d (127.0.98.2:35525)
I20260620 14:20:50.491693 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:20:50.498695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40575
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:50.615002 6832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:50.615338 6832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:50.615424 6832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:50.619654 6832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:50.619741 6832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:50.619877 6832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:50.625862 6832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:50.627498 6832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:50.629040 6832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:20:50.640476 6832 server_base.cc:1034] running on GCE node
W20260620 14:20:50.640429 6837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:50.640455 6838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:50.640689 6840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:50.641184 6832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:50.641865 6832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:50.643086 6832 hybrid_clock.cc:648] HybridClock initialized: now 1781965250643069 us; error 29 us; skew 500 ppm
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:50.645893 6832 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:20:50.646727 6832 webserver.cc:460] Webserver started at http://127.0.98.3:34659/ using document root <none> and password file <none>
I20260620 14:20:50.647245 6832 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:50.647301 6832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:50.647466 6832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:50.649588 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "f925011b90ff4a7e928050da17e12b4a"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "2a15e400ca65b736cebb10e8fa4829df"
server_key_iv: "672302de303085f6634f3ceffe42f943"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.650135 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f925011b90ff4a7e928050da17e12b4a"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "2a15e400ca65b736cebb10e8fa4829df"
server_key_iv: "672302de303085f6634f3ceffe42f943"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.653939 6832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 14:20:50.656378 6847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.657639 6832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:50.657765 6832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "f925011b90ff4a7e928050da17e12b4a"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "2a15e400ca65b736cebb10e8fa4829df"
server_key_iv: "672302de303085f6634f3ceffe42f943"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.657873 6832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:50.682080 6832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:50.685143 6832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:50.685365 6832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:50.685990 6832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:50.686833 6832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:50.686879 6832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.686950 6832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:50.686990 6832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.696331 6832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33965
I20260620 14:20:50.696350 6960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33965 every 8 connection(s)
I20260620 14:20:50.697280 6832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:50.705612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 6832
I20260620 14:20:50.705731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:50.706097 392 external_mini_cluster.cc:1428] Setting key 003fce2ae04f9d1ce4913ac2d06203f5
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:50.711140 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.699104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33709 (local address 127.0.98.62:40575)
0620 14:20:50.699237 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:50.699240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:50.700123 (+ 883us) server_negotiation.cc:407] Connection header received
0620 14:20:50.701390 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:50.701393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:50.701438 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:50.701515 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:50.702637 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.703151 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.704249 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.704454 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.704530 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.707320 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:50.707324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:50.707326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:50.707350 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:50.708919 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.709564 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.709570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.709572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.709661 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.710012 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.710017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.710018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.710207 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:50.710355 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:50.710847 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 14:20:50.710979 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:20:50.712073 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.699500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:33709)
0620 14:20:50.699982 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:50.700022 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:50.701197 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:50.701671 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:50.701678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:50.702038 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:50.702514 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.702527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.703270 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.703272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.704154 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.704161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.704577 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.704579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.704712 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.705218 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:50.705232 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:50.707149 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:50.709021 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.709026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.709038 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.709412 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.709762 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.709766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.709768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.709916 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.710335 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:50.710342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:50.710613 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 14:20:50.710809 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 14:20:50.711149 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":110,"threads_started":1}
I20260620 14:20:50.713220 6961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40575
I20260620 14:20:50.713488 6961 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:50.713984 6961 heartbeater.cc:511] Master 127.0.98.62:40575 requested a full tablet report, sending...
I20260620 14:20:50.715075 6502 ts_manager.cc:194] Registered new tserver with Master: f925011b90ff4a7e928050da17e12b4a (127.0.98.3:33965)
I20260620 14:20:50.715679 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33709
I20260620 14:20:50.720093 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 14:20:50.747013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40949
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40575,127.0.98.61:40949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:50.858683 6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:50.858970 6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:50.859009 6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:50.863161 6966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:50.863224 6966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:50.863245 6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:50.863261 6966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:50.863276 6966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:50.868566 6966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40575,127.0.98.61:40949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
W20260620 14:20:50.868762 6966 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40575,127.0.98.61:40949'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:20:50.869611 6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:50.870591 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:50.879128 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:50.879334 6966 server_base.cc:1034] running on GCE node
W20260620 14:20:50.879148 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:50.879128 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:50.880107 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:50.881240 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:50.882408 6966 hybrid_clock.cc:648] HybridClock initialized: now 1781965250882393 us; error 29 us; skew 500 ppm
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:50.885241 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:20:50.886207 6966 webserver.cc:460] Webserver started at http://127.0.98.61:44545/ using document root <none> and password file <none>
I20260620 14:20:50.886780 6966 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:50.886871 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:50.887092 6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:50.889230 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data/instance:
uuid: "edf297fef1cf4ba788d49e4ada1c5dcc"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "5b49f0b30697739c29efcd0572fb8dfb"
server_key_iv: "4bcf5161494d640f2702b6968ea5a71b"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.889799 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal/instance:
uuid: "edf297fef1cf4ba788d49e4ada1c5dcc"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "5b49f0b30697739c29efcd0572fb8dfb"
server_key_iv: "4bcf5161494d640f2702b6968ea5a71b"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.893491 6966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:20:50.896008 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:50.897311 6966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:50.897473 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal
uuid: "edf297fef1cf4ba788d49e4ada1c5dcc"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "5b49f0b30697739c29efcd0572fb8dfb"
server_key_iv: "4bcf5161494d640f2702b6968ea5a71b"
server_key_version: "encryptionkey@0"
I20260620 14:20:50.897589 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:50.936642 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:50.940284 6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:50.940523 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:50.948021 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40949
I20260620 14:20:50.948334 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40949 every 8 connection(s)
I20260620 14:20:50.949184 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:20:50.952252 7034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:50.953320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 6966
I20260620 14:20:50.956977 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:50.969167 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.954986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.0.98.61:40949)
0620 14:20:50.955459 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:50.955473 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:50.955511 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:20:50.956260 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:50.956277 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:50.956622 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:50.956985 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:50.957784 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.958839 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.960118 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.960823 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.961008 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.964273 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:50.964279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:50.964291 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:50.964313 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:50.966095 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.966545 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.966552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.966561 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.966668 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.966946 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.966951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.966953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.967358 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:50.967571 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:50.967913 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 14:20:50.968242 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":159,"threads_started":1}
I20260620 14:20:50.969286 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.958806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.0.98.62:40575)
0620 14:20:50.958964 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:50.958970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:50.959303 (+ 333us) server_negotiation.cc:407] Connection header received
0620 14:20:50.959438 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:50.959444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:50.959504 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:50.959588 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:50.960450 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.961217 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.962766 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.963170 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.963302 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.965825 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:50.965831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:50.965834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:50.965867 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:50.967333 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.967838 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.967842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.967844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.967966 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.968283 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.968286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.968288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.968576 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:50.968670 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:50.968989 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 14:20:50.969126 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:20:50.969300 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.958731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50054)
0620 14:20:50.959159 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:50.959214 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:50.959333 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:50.959658 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:50.959664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:50.960183 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:50.960311 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.960317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.961396 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.961399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.962631 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.962645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.963334 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.963337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.963424 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.964020 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:50.964037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:50.965666 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:50.967464 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.967474 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.967481 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.967702 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.968079 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.968082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.968084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.968183 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.968667 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:50.968674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:50.968789 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 14:20:50.968923 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:20:50.969110 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":123,"threads_started":1}
I20260620 14:20:50.971257 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:20:50 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:20:50.979394 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.971754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.0.1:43864)
0620 14:20:50.971916 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:50.971930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:50.972001 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:50.972192 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:50.972194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:50.972316 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:50.972731 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.972736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.973404 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.973407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.974198 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:50.974207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.974622 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.974624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:50.974732 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.975290 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:50.975303 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:50.976560 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:50.978089 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.978091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.978093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.978289 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.978518 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:50.978520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:50.978522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:50.978593 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:50.978908 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:50.978910 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:50.978983 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 14:20:50.979080 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:20:50.979202 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:20:50.979403 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:50.971864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.0.98.61:40949)
0620 14:20:50.971964 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:50.971968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:50.971981 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:20:50.972071 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:50.972073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:50.972105 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:50.972172 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:50.972822 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.973303 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.974308 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:50.974521 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:50.974602 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:50.976685 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:50.976688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:50.976690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:50.976709 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:50.977991 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.978373 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.978376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.978377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.978441 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:50.978664 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:50.978666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:50.978667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:50.978812 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:50.978901 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:50.979127 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:20:50.979252 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":39}
I20260620 14:20:50.982666 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Bootstrap starting.
I20260620 14:20:50.985069 7034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:50.985787 7034 log.cc:824] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:50.987586 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: No bootstrap required, opened a new log
I20260620 14:20:50.990669 7034 raft_consensus.cc:348] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } }
I20260620 14:20:50.990887 7034 raft_consensus.cc:374] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:50.990967 7034 raft_consensus.cc:729] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf297fef1cf4ba788d49e4ada1c5dcc, State: Initialized, Role: FOLLOWER
I20260620 14:20:50.991396 7034 consensus_queue.cc:260] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } }
I20260620 14:20:50.991869 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } } }
I20260620 14:20:50.991987 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:20:50.992606 7034 sys_catalog.cc:564] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:20:50.997582 7051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:20:50.997668 7051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:20:51.000026 6966 master_runner.cc:428] Detected that this master edf297fef1cf4ba788d49e4ada1c5dcc is joining an existing cluster
I20260620 14:20:51.000078 6966 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40949
I20260620 14:20:51.011217 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.002813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.0.98.61:40949)
0620 14:20:51.002909 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.002915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.002928 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:20:51.002957 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.002974 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.003008 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.003091 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.003559 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.004367 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.005419 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.005664 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.005753 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.007117 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.007121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.007124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.007150 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.009213 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.009688 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.009695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.009697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.009776 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.010099 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.010105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.010107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.010284 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.010395 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.010907 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.011063 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":38}
I20260620 14:20:51.011265 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.002171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.0.98.62:40575)
0620 14:20:51.002316 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.002321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.002433 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:20:51.002568 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.002573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.002639 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.002717 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.003355 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.004065 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.005202 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.005565 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.005680 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.007491 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.007495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.007498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.007529 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.009447 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.009953 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.009956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.009958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.010045 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.010381 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.010385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.010386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.010567 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.010641 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.010984 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.011107 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:20:51.011229 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.002492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.0.1:43874)
0620 14:20:51.002767 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.002780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.002873 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.003102 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.003104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.003330 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.003442 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.003449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.004523 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.004526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.005316 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.005322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.005875 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.005879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.005982 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.006535 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.006548 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.006979 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.009329 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.009335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.009337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.009601 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.009899 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.009902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.009907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.010026 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.010676 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.010678 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.010754 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.010866 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.011014 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":114,"threads_started":1}
I20260620 14:20:51.011862 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.002071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50066)
0620 14:20:51.002334 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.002350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.002455 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.002842 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.002845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.003075 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.003216 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.003224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.004169 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.004176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.005076 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.005084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.005693 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.005699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.005780 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.006364 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.006374 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.006979 (+ 605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.009575 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.009578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.009580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.009833 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.010156 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.010161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.010166 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.010275 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.010665 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.010668 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.010770 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.011489 (+ 719us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.011652 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"spinlock_wait_cycles":14848,"thread_start_us":102,"threads_started":1}
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:20:51.026245 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.016302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.0.1:43876)
0620 14:20:51.016681 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.016735 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.016875 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.017170 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.017176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.017625 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.017756 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.017763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.018455 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.018459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.019465 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.019476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.019940 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.019944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.020060 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.020589 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.020604 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.022320 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.024492 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.024503 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.024513 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.024808 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.025081 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.025086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.025088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.025168 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.025605 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.025612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.025782 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.025950 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.026116 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
I20260620 14:20:51.026286 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.016325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.0.98.61:40949)
0620 14:20:51.016565 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.016569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.016821 (+ 252us) server_negotiation.cc:407] Connection header received
0620 14:20:51.017006 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.017008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.017050 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.017113 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.017896 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.018360 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.019583 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.019828 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.019912 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.022469 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.022473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.022476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.022500 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.024174 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.024910 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.024917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.024919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.024994 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.025262 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.025265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.025267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.025455 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.025548 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.026025 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.026136 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260620 14:20:51.027670 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40949: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40949
}
attrs {
promote: true
}
}
dest_uuid: "71a466f760d746b2b42fe1b85085da5f"
cas_config_opid_index: -1
I20260620 14:20:51.028407 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } }
I20260620 14:20:51.030592 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER edf297fef1cf4ba788d49e4ada1c5dcc (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } } }
I20260620 14:20:51.031363 7008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:51.032522 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40949
I20260620 14:20:51.032956 7008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Refusing update from remote peer 71a466f760d746b2b42fe1b85085da5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:20:51.032994 6966 master.cc:537] Master@127.0.98.61:40949 shutting down...
I20260620 14:20:51.033416 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } } }
I20260620 14:20:51.033608 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:51.033802 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
W20260620 14:20:51.036260 6487 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260620 14:20:51.037204 6966 raft_consensus.cc:2229] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:51.037473 6966 raft_consensus.cc:2258] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:51.037563 6966 tablet_replica.cc:330] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: stopping tablet replica
W20260620 14:20:51.037846 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f -> Peer edf297fef1cf4ba788d49e4ada1c5dcc (127.0.98.61:40949): Couldn't send request to peer edf297fef1cf4ba788d49e4ada1c5dcc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:20:51.042932 6966 master.cc:552] Master@127.0.98.61:40949 shutdown complete.
I20260620 14:20:51.042989 6966 master_runner.cc:305] Clearing existing system tablet
I20260620 14:20:51.043218 6966 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:51.043273 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:51.044633 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:51.045521 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.045786 6966 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:51.045912 6966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:51.045945 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal
uuid: "edf297fef1cf4ba788d49e4ada1c5dcc"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "5b49f0b30697739c29efcd0572fb8dfb"
server_key_iv: "4bcf5161494d640f2702b6968ea5a71b"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.048213 6966 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:51.051160 6966 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:20:51.051251 6966 log.cc:1196] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:20:51.051910 6966 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Deleting consensus metadata
I20260620 14:20:51.052129 6966 master_runner.cc:315] Copying system tablet from 127.0.98.62:40575
I20260620 14:20:51.053349 6966 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 14:20:51.054735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35659
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40575,127.0.98.61:40949,127.0.98.60:35659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:20:51.062480 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.054020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.0.98.62:40575)
0620 14:20:51.054152 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.054157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.054444 (+ 287us) server_negotiation.cc:407] Connection header received
0620 14:20:51.054590 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.054594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.054648 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.054730 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.055174 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.055704 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.057009 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.057342 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.057454 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.058857 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.058861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.058864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.058890 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.060674 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.061145 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.061149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.061150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.061252 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.061562 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.061566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.061567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.061763 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.061827 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.062166 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.062295 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:20:51.063216 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.053972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50074)
0620 14:20:51.054325 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.054345 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.054490 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.054745 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.054749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.054948 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.055047 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.055053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.055846 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.055849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.056855 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.056877 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.057581 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.057585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.057679 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.058267 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.058281 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.058690 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.060792 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.060795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.060797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.061041 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.061381 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.061383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.061385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.061464 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.061908 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.061911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.062008 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.062887 (+ 879us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.063031 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":86,"threads_started":1}
I20260620 14:20:51.064040 6522 tablet_copy_service.cc:140] P 71a466f760d746b2b42fe1b85085da5f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer edf297fef1cf4ba788d49e4ada1c5dcc ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50074)
I20260620 14:20:51.064253 6522 tablet_copy_service.cc:161] P 71a466f760d746b2b42fe1b85085da5f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer edf297fef1cf4ba788d49e4ada1c5dcc at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50074: session id = edf297fef1cf4ba788d49e4ada1c5dcc-00000000000000000000000000000000
I20260620 14:20:51.066438 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:20:51.068365 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:20:51.070752 6966 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: tablet copy: Starting download of 0 data blocks...
I20260620 14:20:51.070955 6966 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: tablet copy: Starting download of 1 WAL segments...
I20260620 14:20:51.072880 6966 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:20:51.074730 6522 tablet_copy_service.cc:342] P 71a466f760d746b2b42fe1b85085da5f: Request end of tablet copy session edf297fef1cf4ba788d49e4ada1c5dcc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50074
I20260620 14:20:51.074868 6522 tablet_copy_service.cc:433] P 71a466f760d746b2b42fe1b85085da5f: ending tablet copy session edf297fef1cf4ba788d49e4ada1c5dcc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer edf297fef1cf4ba788d49e4ada1c5dcc
I20260620 14:20:51.082940 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:51.084739 7075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.084967 7078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.084789 7076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:51.088828 6966 server_base.cc:1034] running on GCE node
I20260620 14:20:51.089037 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:51.089326 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:51.090468 6966 hybrid_clock.cc:648] HybridClock initialized: now 1781965251090457 us; error 30 us; skew 500 ppm
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:51.092181 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:20:51.092655 6966 webserver.cc:460] Webserver started at http://127.0.98.61:40579/ using document root <none> and password file <none>
I20260620 14:20:51.092847 6966 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:51.092900 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:51.094205 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:51.095170 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.095376 6966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:51.095436 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal
uuid: "edf297fef1cf4ba788d49e4ada1c5dcc"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "5b49f0b30697739c29efcd0572fb8dfb"
server_key_iv: "4bcf5161494d640f2702b6968ea5a71b"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.095515 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:51.115574 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:51.117884 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:51.122519 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40949
I20260620 14:20:51.122725 7137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40949 every 8 connection(s)
I20260620 14:20:51.123135 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:20:51.124872 7138 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:20:51.125869 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Bootstrap starting.
I20260620 14:20:51.135928 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:20:51.136266 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Bootstrap complete.
I20260620 14:20:51.136899 7138 raft_consensus.cc:348] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } }
I20260620 14:20:51.137038 7138 raft_consensus.cc:729] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: edf297fef1cf4ba788d49e4ada1c5dcc, State: Initialized, Role: LEARNER
I20260620 14:20:51.137142 7138 consensus_queue.cc:260] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } }
I20260620 14:20:51.137501 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } } }
I20260620 14:20:51.137612 7138 sys_catalog.cc:564] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:51.137691 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: This master's current role is: LEARNER
I20260620 14:20:51.140661 7152 catalog_manager.cc:1259] Loaded cluster ID: ccd44411486a4584808cf9a7c46c2a6e
I20260620 14:20:51.140776 7152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: loading cluster ID for follower catalog manager: success
I20260620 14:20:51.142895 7152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: acquiring CA information for follower catalog manager: success
I20260620 14:20:51.144200 7152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:20:51.183863 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:51.184135 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:51.184176 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:51.188205 7074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:51.188269 7074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:51.188289 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:51.188306 7074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:51.188350 7074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:51.193862 7074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40575,127.0.98.61:40949,127.0.98.60:35659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:51.195044 7074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:51.196192 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:51.204594 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.204589 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.204608 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:51.204723 7074 server_base.cc:1034] running on GCE node
I20260620 14:20:51.205444 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:51.206586 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:51.207746 7074 hybrid_clock.cc:648] HybridClock initialized: now 1781965251207727 us; error 34 us; skew 500 ppm
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:51.210711 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:20:51.211542 7074 webserver.cc:460] Webserver started at http://127.0.98.60:35193/ using document root <none> and password file <none>
I20260620 14:20:51.212141 7074 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:51.212201 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:51.212365 7074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:51.214325 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data/instance:
uuid: "820e02028d7f478ca3c196b4420e39c9"
format_stamp: "Formatted at 2026-06-20 14:20:51 on dist-test-slave-87nc"
server_key: "35ab7246f6b3cb336fcf6b733aa34863"
server_key_iv: "d19cd84e9d8f6d573bea8234d607284b"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.214826 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal/instance:
uuid: "820e02028d7f478ca3c196b4420e39c9"
format_stamp: "Formatted at 2026-06-20 14:20:51 on dist-test-slave-87nc"
server_key: "35ab7246f6b3cb336fcf6b733aa34863"
server_key_iv: "d19cd84e9d8f6d573bea8234d607284b"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.218559 7074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:20:51.220882 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.222085 7074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:51.222190 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal
uuid: "820e02028d7f478ca3c196b4420e39c9"
format_stamp: "Formatted at 2026-06-20 14:20:51 on dist-test-slave-87nc"
server_key: "35ab7246f6b3cb336fcf6b733aa34863"
server_key_iv: "d19cd84e9d8f6d573bea8234d607284b"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.222272 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:51.242758 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:51.245955 7074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:51.246104 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:51.252661 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35659
I20260620 14:20:51.252950 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35659 every 8 connection(s)
I20260620 14:20:51.253618 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:20:51.256472 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:51.260267 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:20:51.261947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 7074
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:20:51.274607 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.262439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.0.98.62:40575)
0620 14:20:51.262578 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.262584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.263560 (+ 976us) server_negotiation.cc:407] Connection header received
0620 14:20:51.264815 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.264818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.264864 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.264937 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.265877 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.266424 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.267495 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.267716 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.267854 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.270586 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.270592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.270597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.270649 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.272424 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.272955 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.272958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.272960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.273046 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.273425 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.273428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.273430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.273657 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.273728 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.274296 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.274431 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:20:51.275373 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.262769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50080)
0620 14:20:51.263411 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.263456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.264600 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.265008 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.265013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.265444 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.265718 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.265729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.266544 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.266548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.267361 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.267369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.267885 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.267889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.268063 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.268743 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.268764 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.270445 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.272527 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.272535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.272542 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.272854 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.273142 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.273146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.273147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.273320 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.273756 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.273766 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.274054 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.274234 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.274553 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"spinlock_wait_cycles":8320,"thread_start_us":145,"threads_started":1}
I20260620 14:20:51.275381 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.262850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.0.98.60:35659)
0620 14:20:51.263347 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.263363 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.263398 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:20:51.264600 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.264623 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.264858 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.265048 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.265718 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.266703 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.267525 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.268051 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.268151 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.271128 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.271136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.271148 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.271172 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.272972 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.273436 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.273441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.273443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.273513 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.273801 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.273804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.273805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.274050 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.274177 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.274468 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.274605 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":394,"spinlock_wait_cycles":60160,"thread_start_us":163,"threads_started":1}
I20260620 14:20:51.276645 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:20:51.283990 6693 heartbeater.cc:503] Master 127.0.98.62:40575 was elected leader, sending a full tablet report...
I20260620 14:20:51.284590 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.277109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.0.1:43882)
0620 14:20:51.277262 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.277276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.277345 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.277918 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.277920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.278042 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.278124 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.278129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.278970 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.278972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.279493 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.279499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.279952 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.279955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.280023 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.280499 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.280510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.281595 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.283084 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.283088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.283091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.283274 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.283563 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.283566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.283568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.283644 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.284061 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.284064 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.284156 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.284267 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.284404 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 14:20:51.284595 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.277216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.0.98.61:40949)
0620 14:20:51.277710 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.277720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.277732 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:20:51.277767 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.277769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.277807 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.277887 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.278219 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.278883 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.279582 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.279864 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.279941 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.281734 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.281736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.281739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.281758 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.282977 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.283385 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.283389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.283390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.283465 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.283733 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.283738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.283740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.283967 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.284072 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.284326 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.284445 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":126,"threads_started":1}
I20260620 14:20:51.285652 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:20:51.292812 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.286139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35659 (local address 127.0.0.1:37322)
0620 14:20:51.286247 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.286262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.286341 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.286531 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.286533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.286685 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.286749 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.286753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.287385 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.287387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.287947 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.287955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.288335 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.288338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.288389 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.288743 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.288753 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.289858 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.291420 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.291422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.291423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.291589 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.291862 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.291864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.291866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.292102 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.292421 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.292423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.292492 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.292572 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.292680 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:20:51.292912 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.286157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.0.98.60:35659)
0620 14:20:51.286258 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.286262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.286302 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:20:51.286407 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.286409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.286440 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.286501 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.286834 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.287302 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.288037 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.288252 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.288325 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.289971 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.289974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.289976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.289994 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.291324 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.291664 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.291666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.291667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.291729 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.292194 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.292196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.292197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.292340 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.292412 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.292622 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.292730 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":42}
I20260620 14:20:51.296074 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: Bootstrap starting.
I20260620 14:20:51.298321 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:51.299007 7220 log.cc:824] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:51.300788 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: No bootstrap required, opened a new log
I20260620 14:20:51.303568 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } }
I20260620 14:20:51.303789 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:51.303896 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e02028d7f478ca3c196b4420e39c9, State: Initialized, Role: FOLLOWER
I20260620 14:20:51.304307 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } }
I20260620 14:20:51.304741 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } } }
I20260620 14:20:51.304855 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:20:51.305426 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:20:51.310127 7237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:20:51.310205 7237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:20:51.313474 7074 master_runner.cc:428] Detected that this master 820e02028d7f478ca3c196b4420e39c9 is joining an existing cluster
I20260620 14:20:51.313524 7074 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35659
I20260620 14:20:51.328698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 6564
I20260620 14:20:51.334833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32929
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40575
--webserver_port=35651
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40575,127.0.98.61:40949
--tserver_master_addrs=127.0.98.62:40575,127.0.98.61:40949,127.0.98.60:35659 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:20:51.345460 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.315968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35659 (local address 127.0.0.1:37324)
0620 14:20:51.316278 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.316291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.316784 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.317136 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.317139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.317417 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.317514 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.317520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.318528 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.318531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.319406 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.319414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.320064 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.320067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.320140 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.320649 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.320665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.340336 (+ 19671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.342867 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.342870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.342882 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.343126 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.344232 (+ 1106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.344234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.344236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.344344 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.344801 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.344803 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.344946 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.345077 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.345225 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":96,"threads_started":1}
I20260620 14:20:51.346024 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.316843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.0.98.62:40575)
0620 14:20:51.316964 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.316968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.316982 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:20:51.317024 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.317028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.317071 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.317145 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.317772 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.318576 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.320228 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.321119 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.321226 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.341077 (+ 19851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.341081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.341085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.341119 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.343966 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.344590 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.344593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.344595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.344674 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.345045 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.345051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.345056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.345264 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.345345 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.345715 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.345848 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:20:51.346558 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.315856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50090)
0620 14:20:51.316129 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.316143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.316264 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.317255 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.317257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.317522 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.317642 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.317649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.318715 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.318718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.320089 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.320099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.321219 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.321222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.321282 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.321697 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.321707 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.340914 (+ 19207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.344214 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.344219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.344222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.344479 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.344776 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.344780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.344785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.344902 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.345422 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.345424 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.345508 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.346213 (+ 705us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.346369 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"spinlock_wait_cycles":5888,"thread_start_us":122,"threads_started":1}
I20260620 14:20:51.346707 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.316634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.0.98.60:35659)
0620 14:20:51.316956 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.316960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.316976 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:20:51.317001 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.317002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.317032 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.317113 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.317642 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.318405 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.319514 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.319857 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.319932 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.340524 (+ 20592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.340529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.340532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.340564 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.342742 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.344079 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.344083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.344085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.344144 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.344445 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.344449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.344451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.344685 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.344772 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.346474 (+ 1702us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.346573 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260620 14:20:51.349509 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.316316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.0.1:43896)
0620 14:20:51.316596 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.316608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.316819 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.317627 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.317633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.317802 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.317899 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.317906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.318920 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.318923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.319790 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.319798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.320474 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.320476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.320580 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.320964 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.320975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.341318 (+ 20343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.343420 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.343423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.343425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.343657 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.344031 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.344036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.344038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.344126 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.348983 (+ 4857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.348986 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.349085 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.349207 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.349340 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":112,"threads_started":1}
I20260620 14:20:51.349543 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.317265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.0.98.61:40949)
0620 14:20:51.317390 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.317396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.317412 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:20:51.317452 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.317457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.317507 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.317590 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.318033 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.318787 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.319908 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.320248 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.320331 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.341515 (+ 21184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.341521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.341527 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.341565 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.343281 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.343805 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.343813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.343815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.343907 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.348581 (+ 4674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.348587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.348588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.348805 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.348908 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.349280 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.349408 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:20:51.359153 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.351441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.0.98.60:35659)
0620 14:20:51.351541 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.351545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.351598 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:20:51.351750 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.351752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.351871 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.351953 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.352349 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.352876 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.353667 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.353882 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.353987 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.356069 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.356073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.356075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.356098 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.357556 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.357997 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.358002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.358004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.358081 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.358364 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.358368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.358369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.358537 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.358643 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.358910 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.359012 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:20:51.359153 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.351402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35659 (local address 127.0.0.1:37340)
0620 14:20:51.351532 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.351548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.351647 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.351971 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.351974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.352154 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.352235 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.352240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.352990 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.352995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.353549 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.353557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.354000 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.354003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.354103 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.354590 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.354604 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.355917 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.357686 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.357690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.357692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.357896 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.358174 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.358179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.358181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.358268 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.358635 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.358638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.358739 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.358840 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.358988 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:20:51.360507 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35659: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "820e02028d7f478ca3c196b4420e39c9"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35659
}
attrs {
promote: true
}
}
dest_uuid: "71a466f760d746b2b42fe1b85085da5f"
cas_config_opid_index: 5
I20260620 14:20:51.361131 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } }
I20260620 14:20:51.362330 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.363524 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35659
I20260620 14:20:51.363703 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.365201 7194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:51.366623 7194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Refusing update from remote peer 71a466f760d746b2b42fe1b85085da5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:20:51.367347 7074 master.cc:537] Master@127.0.98.60:35659 shutting down...
I20260620 14:20:51.368862 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:51.367197 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:51.370013 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.364027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.0.1:43906)
0620 14:20:51.364125 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.364142 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.364210 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.364752 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.364756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.364775 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:51.364860 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.364866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.365906 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.365909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.367235 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.367244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.369579 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.369582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.369672 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.369723 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.369798 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.369889 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:20:51.370711 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.364278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.0.98.61:40949)
0620 14:20:51.364383 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.364388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.364400 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:20:51.364448 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.364450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.364521 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.364616 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:51.365000 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.365719 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.367369 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.368151 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.368253 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.368407 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.370504 (+ 2097us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.370584 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 14:20:51.370725 6489 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f -> Peer 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60:35659): Couldn't send request to peer 820e02028d7f478ca3c196b4420e39c9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:20:51.372061 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 LEARNER]: Refusing update from remote peer 71a466f760d746b2b42fe1b85085da5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:20:51.372934 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:20:51.374760 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.376741 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.376886 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: This master's current role is: LEARNER
I20260620 14:20:51.376989 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71a466f760d746b2b42fe1b85085da5f. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: true } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.377061 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: This master's current role is: LEARNER
I20260620 14:20:51.377691 7062 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: attempting to promote NON_VOTER edf297fef1cf4ba788d49e4ada1c5dcc to VOTER
I20260620 14:20:51.378123 7062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } }
I20260620 14:20:51.379588 7074 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:51.379818 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 LEARNER]: Refusing update from remote peer 71a466f760d746b2b42fe1b85085da5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:20:51.379880 7074 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:51.379978 7074 tablet_replica.cc:330] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: stopping tablet replica
W20260620 14:20:51.380169 6489 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f -> Peer 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60:35659): Couldn't send request to peer 820e02028d7f478ca3c196b4420e39c9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:20:51.380173 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 14:20:51.381901 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, edf297fef1cf4ba788d49e4ada1c5dcc (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.383217 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.383324 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:51.383347 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.383450 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:51.384155 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, edf297fef1cf4ba788d49e4ada1c5dcc (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.386021 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.386145 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:20:51.397454 7074 master.cc:552] Master@127.0.98.60:35659 shutdown complete.
I20260620 14:20:51.397543 7074 master_runner.cc:305] Clearing existing system tablet
I20260620 14:20:51.397778 7074 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:51.397857 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:51.399036 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:20:51.399936 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.400177 7074 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:51.400309 7074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:51.400353 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal
uuid: "820e02028d7f478ca3c196b4420e39c9"
format_stamp: "Formatted at 2026-06-20 14:20:51 on dist-test-slave-87nc"
server_key: "35ab7246f6b3cb336fcf6b733aa34863"
server_key_iv: "d19cd84e9d8f6d573bea8234d607284b"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.402154 7074 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:51.404132 7074 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:20:51.404237 7074 log.cc:1196] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:20:51.404469 7074 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: Deleting consensus metadata
I20260620 14:20:51.404654 7074 master_runner.cc:315] Copying system tablet from 127.0.98.62:40575
I20260620 14:20:51.405802 7074 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40575
I20260620 14:20:51.415268 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.406471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.0.98.62:40575)
0620 14:20:51.406608 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.406612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.406897 (+ 285us) server_negotiation.cc:407] Connection header received
0620 14:20:51.407079 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.407083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.407132 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.407236 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.407760 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.408524 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.409592 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.409885 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.410015 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.411732 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.411736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.411739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.411794 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.413444 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.413945 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.413949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.413951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.414066 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.414380 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.414384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.414388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.414579 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.414689 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.414976 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.415094 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:20:51.415282 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.406459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50094)
0620 14:20:51.406785 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.406797 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.406937 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.407357 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.407360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.407530 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.407636 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.407641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.408650 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.408657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.409474 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.409484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.410163 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.410166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.410237 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.411235 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.411246 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.411581 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.413553 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.413558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.413560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.413837 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.414163 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.414168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.414170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.414299 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.414676 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.414679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.414780 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.414935 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.415096 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":84,"threads_started":1}
I20260620 14:20:51.416118 6522 tablet_copy_service.cc:140] P 71a466f760d746b2b42fe1b85085da5f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 820e02028d7f478ca3c196b4420e39c9 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50094)
I20260620 14:20:51.416235 6522 tablet_copy_service.cc:161] P 71a466f760d746b2b42fe1b85085da5f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 820e02028d7f478ca3c196b4420e39c9 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50094: session id = 820e02028d7f478ca3c196b4420e39c9-00000000000000000000000000000000
I20260620 14:20:51.417194 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:20:51.418345 7074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:20:51.420428 7074 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: tablet copy: Starting download of 0 data blocks...
I20260620 14:20:51.420625 7074 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: tablet copy: Starting download of 1 WAL segments...
I20260620 14:20:51.421983 7074 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:20:51.423463 6522 tablet_copy_service.cc:342] P 71a466f760d746b2b42fe1b85085da5f: Request end of tablet copy session 820e02028d7f478ca3c196b4420e39c9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50094
I20260620 14:20:51.423560 6522 tablet_copy_service.cc:433] P 71a466f760d746b2b42fe1b85085da5f: ending tablet copy session 820e02028d7f478ca3c196b4420e39c9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 820e02028d7f478ca3c196b4420e39c9
I20260620 14:20:51.430311 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:51.431952 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.432060 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.432220 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:51.436316 7074 server_base.cc:1034] running on GCE node
I20260620 14:20:51.436525 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:51.436857 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:51.438031 7074 hybrid_clock.cc:648] HybridClock initialized: now 1781965251437991 us; error 60 us; skew 500 ppm
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:51.439646 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:20:51.440193 7074 webserver.cc:460] Webserver started at http://127.0.98.60:41169/ using document root <none> and password file <none>
I20260620 14:20:51.440395 7074 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:51.440454 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:51.441581 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:20:51.442539 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.442764 7074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:20:51.442862 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal
uuid: "820e02028d7f478ca3c196b4420e39c9"
format_stamp: "Formatted at 2026-06-20 14:20:51 on dist-test-slave-87nc"
server_key: "35ab7246f6b3cb336fcf6b733aa34863"
server_key_iv: "d19cd84e9d8f6d573bea8234d607284b"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.442950 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:51.459491 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:51.461874 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:51.466885 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35659
I20260620 14:20:51.467420 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:20:51.467933 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35659 every 8 connection(s)
I20260620 14:20:51.469470 7321 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:20:51.470306 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: Bootstrap starting.
I20260620 14:20:51.478996 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:20:51.479291 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: Bootstrap complete.
I20260620 14:20:51.479803 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } }
I20260620 14:20:51.479954 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 820e02028d7f478ca3c196b4420e39c9, State: Initialized, Role: LEARNER
I20260620 14:20:51.480072 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } }
I20260620 14:20:51.480413 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.480573 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:51.480584 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:20:51.484305 7335 catalog_manager.cc:1259] Loaded cluster ID: ccd44411486a4584808cf9a7c46c2a6e
I20260620 14:20:51.484432 7335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: loading cluster ID for follower catalog manager: success
I20260620 14:20:51.485937 7335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: acquiring CA information for follower catalog manager: success
I20260620 14:20:51.487134 7335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:20:51.493790 6827 heartbeater.cc:503] Master 127.0.98.62:40575 was elected leader, sending a full tablet report...
W20260620 14:20:51.493649 7246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:51.493979 7246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:51.494055 7246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:51.499980 7246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:51.500062 7246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:51.500152 7246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:51.507933 7246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35651
--tserver_master_addrs=127.0.98.62:40575,127.0.98.61:40949,127.0.98.60:35659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:51.509356 7246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:51.510738 7246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:51.522102 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.522053 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.522040 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:51.522493 7246 server_base.cc:1034] running on GCE node
I20260620 14:20:51.522881 7246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:51.523551 7246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:51.524736 7246 hybrid_clock.cc:648] HybridClock initialized: now 1781965251524706 us; error 48 us; skew 500 ppm
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:51.527508 7246 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:20:51.528409 7246 webserver.cc:460] Webserver started at http://127.0.98.1:35651/ using document root <none> and password file <none>
I20260620 14:20:51.528956 7246 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:51.529069 7246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:51.532835 7246 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 14:20:51.534783 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.536007 7246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:51.536131 7246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "e15ef7c8dfbf4028bd99881804e1bc13"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "8f086aaf413f58514500b5390cb90f1c"
server_key_iv: "99a7a0ee05fc7f03165bbdc17b62edf3"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.536662 7246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:51.566707 7246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:51.570343 7246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:51.570545 7246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:51.571151 7246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:51.572082 7246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:51.572129 7246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.572196 7246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:51.572237 7246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.581934 7246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32929
I20260620 14:20:51.582012 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32929 every 8 connection(s)
I20260620 14:20:51.583009 7246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:51.592711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 7246
I20260620 14:20:51.592921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 6698
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:20:51.600927 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.586855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54669 (local address 127.0.98.62:40575)
0620 14:20:51.587020 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.587025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.587141 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:20:51.587411 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.587414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.587474 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.587533 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.589034 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.589833 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.591120 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.591496 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.591589 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.595927 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.595933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.595939 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.595974 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.598877 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.599392 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.599395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.599397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.599496 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.599856 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.599859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.599861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.600259 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.600365 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.600643 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.600762 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 14:20:51.600943 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.584901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56813 (local address 127.0.98.61:40949)
0620 14:20:51.585041 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.585047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.586354 (+ 1307us) server_negotiation.cc:407] Connection header received
0620 14:20:51.587413 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.587416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.587476 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.587540 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.589014 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.589530 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.591164 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.591454 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.591546 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.595903 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.595907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.595909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.595935 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.598792 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.599392 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.599395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.599397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.599496 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.599856 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.599859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.599861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.600054 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.600109 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.600695 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.600813 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:20:51.601716 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.587347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57179 (local address 127.0.98.60:35659)
0620 14:20:51.587602 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.587616 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.587844 (+ 228us) server_negotiation.cc:407] Connection header received
0620 14:20:51.588018 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.588023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.588082 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.588145 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.589034 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.590072 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.590958 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.591289 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.591398 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.595903 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.595907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.595909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.595935 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.597771 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.598428 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.598434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.598439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.598529 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.598950 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.598956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.598961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.599218 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.599317 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.601413 (+ 2096us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.601557 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260620 14:20:51.601958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35525
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40575
--webserver_port=38995
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40575,127.0.98.61:40949
--tserver_master_addrs=127.0.98.62:40575,127.0.98.61:40949,127.0.98.60:35659 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:20:51.601972 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.585373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.98.1:56813)
0620 14:20:51.586205 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.586247 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.587169 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.587911 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.587917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.588351 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.588871 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.588884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.589666 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.589669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.591008 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.591018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.591664 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.591666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.591749 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.592724 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.592739 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.595698 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.598919 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.598925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.598928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.599279 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.599602 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.599605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.599607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.599722 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.600428 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.600431 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.600513 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.601122 (+ 609us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.601347 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":170,"spinlock_wait_cycles":18816,"thread_start_us":515,"threads_started":1}
I20260620 14:20:51.602257 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.586709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:54669)
0620 14:20:51.587032 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.587045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.587168 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.587910 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.587919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.588408 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.588896 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.588902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.589941 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.589944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.591008 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.591018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.591716 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.591720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.591839 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.592614 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.592632 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.595736 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.598981 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.598983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.598985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.599278 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.599603 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.599605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.599607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.599713 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.600356 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.600359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.600454 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.601122 (+ 668us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.601399 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":277,"spinlock_wait_cycles":44544,"thread_start_us":100,"threads_started":1}
I20260620 14:20:51.602715 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.587255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35659 (local address 127.0.98.1:57179)
0620 14:20:51.587741 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.587754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.587910 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.588299 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.588302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.588488 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.588893 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.588902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.590175 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.590179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.590834 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.590842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.591557 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.591563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.591747 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.592558 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.592581 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.595698 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.597911 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.597920 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.597938 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.598304 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.598632 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.598637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.598642 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.598821 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.599374 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.599381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.599701 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.599944 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.600302 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":139,"thread_start_us":124,"threads_started":1}
I20260620 14:20:51.603449 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40949
I20260620 14:20:51.603690 7466 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:51.603840 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40575
I20260620 14:20:51.603955 7465 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:51.604313 7465 heartbeater.cc:511] Master 127.0.98.62:40575 requested a full tablet report, sending...
I20260620 14:20:51.604313 7466 heartbeater.cc:511] Master 127.0.98.61:40949 requested a full tablet report, sending...
I20260620 14:20:51.604467 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35659
I20260620 14:20:51.604532 7464 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:51.604655 7464 heartbeater.cc:511] Master 127.0.98.60:35659 requested a full tablet report, sending...
I20260620 14:20:51.605556 6502 ts_manager.cc:194] Re-registered known tserver with Master: e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929)
I20260620 14:20:51.605731 7102 ts_manager.cc:194] Registered new tserver with Master: e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929)
I20260620 14:20:51.605803 7285 ts_manager.cc:194] Registered new tserver with Master: e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929)
I20260620 14:20:51.606266 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54669
W20260620 14:20:51.714804 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:51.715072 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:51.715143 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:20:51.717806 6961 heartbeater.cc:503] Master 127.0.98.62:40575 was elected leader, sending a full tablet report...
W20260620 14:20:51.719442 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:51.719521 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:51.719604 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:51.724776 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38995
--tserver_master_addrs=127.0.98.62:40575,127.0.98.61:40949,127.0.98.60:35659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:51.725718 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:51.726718 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:51.736024 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.736032 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.736058 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:51.736330 7472 server_base.cc:1034] running on GCE node
I20260620 14:20:51.736676 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:51.737293 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:51.738468 7472 hybrid_clock.cc:648] HybridClock initialized: now 1781965251738458 us; error 31 us; skew 500 ppm
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:51.741254 7472 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:20:51.742106 7472 webserver.cc:460] Webserver started at http://127.0.98.2:38995/ using document root <none> and password file <none>
I20260620 14:20:51.742683 7472 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:51.742792 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:51.746605 7472 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:51.748522 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.749719 7472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:51.749873 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "96f5529d6fb143cdae32db52d94cc28d"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "85a51f50cd18904e2b958d0b61e490fb"
server_key_iv: "b9265060874b117f8e9622c5976ed21d"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.750406 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:51.765380 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.761803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35659 (local address 127.0.0.1:37346)
0620 14:20:51.761934 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.761950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.762038 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.762386 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.762388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.762403 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:51.762482 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.762487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.763456 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.763458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.764241 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.764250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.765088 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.765090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.765163 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.765181 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.765226 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.765275 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:20:51.765424 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.761883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.0.98.60:35659)
0620 14:20:51.762003 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.762008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.762024 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:20:51.762133 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.762136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.762190 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.762316 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:51.762586 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.763353 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.764363 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.764983 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.765078 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.765207 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.765237 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.765296 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:20:51.765987 7287 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 14:20:51.766388 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71a466f760d746b2b42fe1b85085da5f. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:51.766551 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:20:51.780689 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:51.783730 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:51.783921 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:51.784457 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:51.785274 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:51.785321 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.785360 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:51.785385 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.794378 7472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35525
I20260620 14:20:51.794401 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35525 every 8 connection(s)
I20260620 14:20:51.795444 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:51.800010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 7472
I20260620 14:20:51.800197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 6832
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:51.808113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33965
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40575
--webserver_port=34659
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40575,127.0.98.61:40949
--tserver_master_addrs=127.0.98.62:40575,127.0.98.61:40949,127.0.98.60:35659 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:20:51.812103 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.798897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60555 (local address 127.0.98.61:40949)
0620 14:20:51.799015 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.799020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.799132 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:20:51.799792 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.799796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.799844 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.799950 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.801529 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.802342 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.803855 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.804063 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.804178 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.807453 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.807456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.807458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.807481 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.809422 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.810207 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.810211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.810213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.810313 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.810757 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.810761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.810763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.810984 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.811048 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.811808 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.811935 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:20:51.812103 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.797380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56275 (local address 127.0.98.62:40575)
0620 14:20:51.797517 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.797521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.798665 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 14:20:51.799608 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.799615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.799665 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.799740 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.801347 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.801937 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.803651 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.804144 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.804260 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.807209 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.807216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.807222 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.807259 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.809422 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.810207 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.810211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.810213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.810313 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.810757 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.810761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.810762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.810973 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.811046 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.811809 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.811935 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:20:51.813862 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.797779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:56275)
0620 14:20:51.798572 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.798586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.799316 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.799901 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.799910 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.800474 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.801177 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.801197 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.802076 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.802080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.803518 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.803528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.804400 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.804407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.804538 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.805118 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.805138 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.807040 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.809568 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.809575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.809592 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.810090 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.810419 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.810422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.810424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.810644 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.811130 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.811140 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.811489 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.812679 (+ 1190us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.812865 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":132,"spinlock_wait_cycles":28672,"thread_start_us":567,"threads_started":1}
I20260620 14:20:51.814199 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.798774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.98.2:60555)
0620 14:20:51.799039 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.799051 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.799316 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.799971 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.799974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.801263 (+ 1289us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.801401 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.801408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.802463 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.802466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.803518 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.803528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.804211 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.804223 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.804389 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.804968 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.804994 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.807315 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.809568 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.809575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.809592 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.810090 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.810427 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.810430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.810431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.810644 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.811403 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.811406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.811489 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.812367 (+ 878us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.812734 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":54,"spinlock_wait_cycles":37248,"thread_start_us":117,"threads_started":1}
I20260620 14:20:51.815495 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40575
I20260620 14:20:51.815722 7602 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:51.815838 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40949
I20260620 14:20:51.815901 7603 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:51.816454 7603 heartbeater.cc:511] Master 127.0.98.61:40949 requested a full tablet report, sending...
I20260620 14:20:51.817123 7602 heartbeater.cc:511] Master 127.0.98.62:40575 requested a full tablet report, sending...
I20260620 14:20:51.817258 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.797421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33607 (local address 127.0.98.60:35659)
0620 14:20:51.797538 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.797543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.798433 (+ 890us) server_negotiation.cc:407] Connection header received
0620 14:20:51.799486 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.799489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.799547 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.799632 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.808160 (+ 8528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.809009 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.810125 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.810509 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.810619 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.813316 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:51.813323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:51.813330 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:51.813367 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:51.815287 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.815864 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.815868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.815872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.815962 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:51.816307 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:51.816311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:51.816312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:51.816534 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:51.816634 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:51.816984 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 14:20:51.817108 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:20:51.817374 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.797779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35659 (local address 127.0.98.2:33607)
0620 14:20:51.798259 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.798308 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.799293 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.807644 (+ 8351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.807650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.807841 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.807972 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.807979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.809206 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.809212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.809999 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.810009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.810806 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.810812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.810939 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:51.811464 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:51.811493 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:51.813140 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:51.815432 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.815438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.815441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.815714 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.816074 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:51.816079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:51.816083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:51.816187 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:51.816646 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:51.816651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:51.816770 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 14:20:51.816901 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:20:51.817074 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"spinlock_wait_cycles":768,"thread_start_us":138,"threads_started":1}
I20260620 14:20:51.818007 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35659
I20260620 14:20:51.818125 7601 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:51.818209 6502 ts_manager.cc:194] Re-registered known tserver with Master: 96f5529d6fb143cdae32db52d94cc28d (127.0.98.2:35525)
I20260620 14:20:51.818359 7601 heartbeater.cc:511] Master 127.0.98.60:35659 requested a full tablet report, sending...
I20260620 14:20:51.818342 7102 ts_manager.cc:194] Registered new tserver with Master: 96f5529d6fb143cdae32db52d94cc28d (127.0.98.2:35525)
I20260620 14:20:51.818923 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56275
I20260620 14:20:51.818976 7285 ts_manager.cc:194] Registered new tserver with Master: 96f5529d6fb143cdae32db52d94cc28d (127.0.98.2:35525)
W20260620 14:20:51.923565 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:51.923857 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:51.923900 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:51.928047 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:51.928108 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:51.928167 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:51.933270 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34659
--tserver_master_addrs=127.0.98.62:40575,127.0.98.61:40949,127.0.98.60:35659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:51.934237 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:51.935279 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:51.945044 7616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.945132 7614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:51.945044 7613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:51.945173 7608 server_base.cc:1034] running on GCE node
I20260620 14:20:51.945700 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:51.946323 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:51.947479 7608 hybrid_clock.cc:648] HybridClock initialized: now 1781965251947441 us; error 52 us; skew 500 ppm
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:51.950205 7608 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:20:51.951042 7608 webserver.cc:460] Webserver started at http://127.0.98.3:34659/ using document root <none> and password file <none>
I20260620 14:20:51.951599 7608 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:51.951704 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:51.955557 7608 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:51.957589 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.958751 7608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:51.958866 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "f925011b90ff4a7e928050da17e12b4a"
format_stamp: "Formatted at 2026-06-20 14:20:50 on dist-test-slave-87nc"
server_key: "2a15e400ca65b736cebb10e8fa4829df"
server_key_iv: "672302de303085f6634f3ceffe42f943"
server_key_version: "encryptionkey@0"
I20260620 14:20:51.959374 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:51.976181 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:51.978852 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:51.979044 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:51.979610 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:51.980481 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:51.980527 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.980613 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:51.980646 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:51.989977 7608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33965
I20260620 14:20:51.990254 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33965 every 8 connection(s)
I20260620 14:20:51.991147 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:51.996107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 7608
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:20:51 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:20:52 dist-test-slave-87nc krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:20:52.007297 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.995308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37403 (local address 127.0.98.61:40949)
0620 14:20:51.995462 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.995466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.996096 (+ 630us) server_negotiation.cc:407] Connection header received
0620 14:20:51.996323 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.996327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.996377 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.996431 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.997092 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.997842 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.999961 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.000287 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.000361 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.002995 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.002999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.003001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.003032 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.004798 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.005443 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.005447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.005449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.005545 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.005981 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.005985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.005986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.006201 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.006269 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.006999 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.007129 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:20:52.008913 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.995208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.98.3:37403)
0620 14:20:51.996006 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.996021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.996202 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.996584 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.996586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.996800 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.996941 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.996949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.998035 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.998041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.999828 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.999836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.000507 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.000511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.000616 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.001098 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:52.001112 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:52.002800 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:52.004930 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:52.004938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:52.004955 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:52.005308 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:52.005658 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:52.005663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:52.005665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:52.005821 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:52.006366 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:52.006372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:52.006675 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 14:20:52.007455 (+ 780us) client_negotiation.cc:240] Negotiation successful
0620 14:20:52.007801 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":782,"thread_start_us":73,"threads_started":1}
I20260620 14:20:52.010318 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40949
I20260620 14:20:52.010586 7739 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:52.010773 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.994458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43193 (local address 127.0.98.62:40575)
0620 14:20:51.994588 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.994592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.994605 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:20:51.995321 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.995325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.995379 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.995432 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.997074 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.997821 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.999534 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.999907 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.999980 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.004243 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.004249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.004256 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.004286 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.006128 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.008001 (+ 1873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.008007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.008012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.008392 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.009766 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.009772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.009775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.010020 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.010128 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.010456 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.010593 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:20:52.011251 7739 heartbeater.cc:511] Master 127.0.98.61:40949 requested a full tablet report, sending...
I20260620 14:20:52.011335 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.993427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:43193)
0620 14:20:51.994500 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.994515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.995207 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.995853 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.995862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.996314 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.996922 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.996940 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.998007 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.998009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:51.999401 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.999411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.000158 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.000161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.000338 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.000955 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:52.000974 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:52.004079 (+ 3105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:52.006846 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:52.006849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:52.006858 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:52.007273 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:52.009537 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:52.009540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:52.009542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:52.009648 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:52.010173 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:52.010177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:52.010270 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:20:52.010938 (+ 668us) client_negotiation.cc:240] Negotiation successful
0620 14:20:52.011088 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":47,"spinlock_wait_cycles":38144,"thread_start_us":117,"threads_started":1}
I20260620 14:20:52.013937 7102 ts_manager.cc:194] Registered new tserver with Master: f925011b90ff4a7e928050da17e12b4a (127.0.98.3:33965)
I20260620 14:20:52.014117 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.001784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.0.98.60:35659)
0620 14:20:52.002443 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.002448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.004389 (+ 1941us) server_negotiation.cc:407] Connection header received
0620 14:20:52.004586 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.004591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.004640 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.004704 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:52.005678 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.006674 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.008089 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.008407 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.008477 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.010452 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.010456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.010458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.010480 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.012363 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.012807 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.012811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.012812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.012899 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.013201 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.013204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.013206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.013399 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.013496 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.013821 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.013960 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":76,"threads_started":1}
I20260620 14:20:52.014835 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40575
I20260620 14:20:52.014925 7738 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:52.015103 7738 heartbeater.cc:511] Master 127.0.98.62:40575 requested a full tablet report, sending...
I20260620 14:20:52.015537 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.002001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.0.98.62:40575)
0620 14:20:52.002880 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.002886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.002900 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:20:52.002939 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.002943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.002991 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.003067 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:52.004664 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.005559 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.006939 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.007577 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.008146 (+ 569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.010214 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.010218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.010222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.010251 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.012728 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.013274 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.013277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.013279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.013374 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.013721 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.013724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.013726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.013933 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.015004 (+ 1071us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.015092 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.015312 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":69,"threads_started":1}
I20260620 14:20:52.015925 6502 ts_manager.cc:194] Re-registered known tserver with Master: f925011b90ff4a7e928050da17e12b4a (127.0.98.3:33965)
I20260620 14:20:52.017061 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43193
I20260620 14:20:52.019073 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.992971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45159 (local address 127.0.98.60:35659)
0620 14:20:51.993096 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:51.993101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:51.994123 (+ 1022us) server_negotiation.cc:407] Connection header received
0620 14:20:51.998344 (+ 4221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:51.998349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:51.998409 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:51.998466 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:51.999018 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.999826 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.001418 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.001775 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.001855 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.012301 (+ 10446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.012306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.012309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.012338 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.017021 (+ 4683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.017576 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.017581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.017584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.017680 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.018061 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.018067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.018071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.018324 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.018402 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.018779 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.018910 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:20:52.019827 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:51.993427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35659 (local address 127.0.98.3:45159)
0620 14:20:51.993963 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:51.994029 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:51.995034 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:51.998608 (+ 3574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:51.998611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:51.998780 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:51.998889 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:51.998895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:51.999960 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:51.999963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.001259 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.001271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.002225 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.002232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.002319 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.003086 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:52.003100 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:52.010035 (+ 6935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:52.017173 (+ 7138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:52.017177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:52.017179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:52.017446 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:52.017814 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:52.017817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:52.017819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:52.017937 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:52.018484 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:52.018487 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:52.018580 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:20:52.019228 (+ 648us) client_negotiation.cc:240] Negotiation successful
0620 14:20:52.019390 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":136,"threads_started":1}
I20260620 14:20:52.020507 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35659
I20260620 14:20:52.020589 7737 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:52.020779 7737 heartbeater.cc:511] Master 127.0.98.60:35659 requested a full tablet report, sending...
I20260620 14:20:52.021364 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.002745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.0.98.61:40949)
0620 14:20:52.002978 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.002984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.009237 (+ 6253us) server_negotiation.cc:407] Connection header received
0620 14:20:52.009274 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.009277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.009342 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.009416 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:52.012131 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.012927 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.014003 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.014645 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.014736 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.017700 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.017704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.017706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.017728 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.019506 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.020050 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.020053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.020055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.020138 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.020435 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.020438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.020439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.020642 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.020754 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.021071 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.021200 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":65,"threads_started":1}
I20260620 14:20:52.021497 7285 ts_manager.cc:194] Registered new tserver with Master: f925011b90ff4a7e928050da17e12b4a (127.0.98.3:33965)
I20260620 14:20:52.019356 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:52.023401 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:52.025964 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: true } } }
I20260620 14:20:52.026095 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:52.029767 7756 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: attempting to promote NON_VOTER 820e02028d7f478ca3c196b4420e39c9 to VOTER
I20260620 14:20:52.030114 7756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } }
I20260620 14:20:52.031352 7287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 LEARNER]: Refusing update from remote peer 71a466f760d746b2b42fe1b85085da5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:20:52.031349 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Refusing update from remote peer 71a466f760d746b2b42fe1b85085da5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:20:52.031701 7755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 14:20:52.031850 7756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 14:20:52.033670 7249 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:52.034253 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:52.034974 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:52.035086 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:52.035228 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:52.035296 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:52.035416 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:52.035494 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:52.035969 7287 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:52.036569 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:52.036792 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:20:52.038110 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71a466f760d746b2b42fe1b85085da5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:52.038232 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:20:52.045142 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.039139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32929 (local address 127.0.0.1:42866)
0620 14:20:52.039309 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:52.039325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:52.039429 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:52.041043 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:52.041045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:52.041060 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:52.041141 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.041145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.042482 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.042484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.043552 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.043561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.044705 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.044709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.044801 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.044829 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:20:52.044922 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:20:52.044990 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":1}
I20260620 14:20:52.045202 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.039942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.0.98.1:32929)
0620 14:20:52.040543 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.040562 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.040584 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:20:52.040662 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.040667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.040921 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.041060 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:52.041261 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.042383 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.043707 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.044580 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.044699 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.044849 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.044953 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.045042 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":95,"threads_started":1}
I20260620 14:20:52.046046 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.039211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33965 (local address 127.0.0.1:50984)
0620 14:20:52.040039 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:52.040056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:52.040264 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:52.042126 (+ 1862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:52.042128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:52.042143 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:52.042209 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.042213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.043341 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.043345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.044601 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.044612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.045658 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.045661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.045743 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.045764 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:20:52.045854 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:20:52.045921 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":129,"threads_started":1}
I20260620 14:20:52.046113 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.039481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.0.98.3:33965)
0620 14:20:52.041698 (+ 2217us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.041710 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.041733 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:20:52.041806 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.041812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.041994 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.042154 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:52.042278 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.043238 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.044722 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.045544 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.045659 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.045789 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.045887 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.045972 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2095,"thread_start_us":118,"threads_started":1}
I20260620 14:20:52.048908 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.039602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35525 (local address 127.0.0.1:59564)
0620 14:20:52.041317 (+ 1715us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:52.041332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:52.041427 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:52.044056 (+ 2629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:52.044059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:52.044077 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:52.044186 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.044194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.046442 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.046446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.047575 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.047588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.048533 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.048536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.048618 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.048640 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:20:52.048714 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:20:52.048780 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"thread_start_us":94,"threads_started":1}
I20260620 14:20:52.048956 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.039754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.0.98.2:35525)
0620 14:20:52.043548 (+ 3794us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.043565 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.043589 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:20:52.043679 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.043685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.043911 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.044067 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:52.044289 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.045540 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.047713 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.048412 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.048534 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.048660 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.048740 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.048825 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3679,"thread_start_us":119,"threads_started":1}
I20260620 14:20:52.049014 7398 tablet_service.cc:1467] Processing CreateTablet for tablet f2350a935b41465bae6291def9f26346 (DEFAULT_TABLE table=test-table [id=75ef14fa0bb24d229e2da29ed80b5de1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:52.050171 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2350a935b41465bae6291def9f26346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:52.049803 7671 tablet_service.cc:1467] Processing CreateTablet for tablet f2350a935b41465bae6291def9f26346 (DEFAULT_TABLE table=test-table [id=75ef14fa0bb24d229e2da29ed80b5de1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:52.050885 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2350a935b41465bae6291def9f26346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:52.050913 7535 tablet_service.cc:1467] Processing CreateTablet for tablet f2350a935b41465bae6291def9f26346 (DEFAULT_TABLE table=test-table [id=75ef14fa0bb24d229e2da29ed80b5de1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:52.051623 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2350a935b41465bae6291def9f26346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:52.057801 7770 tablet_bootstrap.cc:492] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: Bootstrap starting.
I20260620 14:20:52.059428 7772 tablet_bootstrap.cc:492] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: Bootstrap starting.
I20260620 14:20:52.060114 7771 tablet_bootstrap.cc:492] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: Bootstrap starting.
I20260620 14:20:52.060662 7770 tablet_bootstrap.cc:654] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:52.061365 7770 log.cc:824] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:52.061676 7772 tablet_bootstrap.cc:654] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:52.062196 7771 tablet_bootstrap.cc:654] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:52.062552 7772 log.cc:824] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:52.063014 7771 log.cc:824] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:52.063519 7770 tablet_bootstrap.cc:492] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: No bootstrap required, opened a new log
I20260620 14:20:52.063714 7770 ts_tablet_manager.cc:1397] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 14:20:52.064646 7772 tablet_bootstrap.cc:492] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: No bootstrap required, opened a new log
I20260620 14:20:52.064847 7772 ts_tablet_manager.cc:1397] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:20:52.065801 7771 tablet_bootstrap.cc:492] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: No bootstrap required, opened a new log
I20260620 14:20:52.065979 7771 ts_tablet_manager.cc:1397] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:20:52.067842 7770 raft_consensus.cc:348] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:52.068136 7770 raft_consensus.cc:374] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:52.068190 7770 raft_consensus.cc:729] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f5529d6fb143cdae32db52d94cc28d, State: Initialized, Role: FOLLOWER
I20260620 14:20:52.068629 7770 consensus_queue.cc:260] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:52.069306 7602 heartbeater.cc:503] Master 127.0.98.62:40575 was elected leader, sending a full tablet report...
I20260620 14:20:52.070091 7771 raft_consensus.cc:348] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:52.070370 7771 raft_consensus.cc:374] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:52.070423 7771 raft_consensus.cc:729] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f925011b90ff4a7e928050da17e12b4a, State: Initialized, Role: FOLLOWER
I20260620 14:20:52.070868 7771 consensus_queue.cc:260] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:52.071934 7771 ts_tablet_manager.cc:1428] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:20:52.070097 7772 raft_consensus.cc:348] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:52.072427 7772 raft_consensus.cc:374] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:52.072485 7772 raft_consensus.cc:729] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15ef7c8dfbf4028bd99881804e1bc13, State: Initialized, Role: FOLLOWER
I20260620 14:20:52.072983 7772 consensus_queue.cc:260] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:52.073139 7738 heartbeater.cc:503] Master 127.0.98.62:40575 was elected leader, sending a full tablet report...
I20260620 14:20:52.073222 7770 ts_tablet_manager.cc:1428] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260620 14:20:52.073999 7772 ts_tablet_manager.cc:1428] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260620 14:20:52.075223 7465 heartbeater.cc:503] Master 127.0.98.62:40575 was elected leader, sending a full tablet report...
W20260620 14:20:52.084686 7467 tablet.cc:2369] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:52.167470 7776 raft_consensus.cc:482] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:52.167685 7776 raft_consensus.cc:504] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:52.168978 7776 leader_election.cc:288] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f925011b90ff4a7e928050da17e12b4a (127.0.98.3:33965), e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929)
I20260620 14:20:52.174077 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.169372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35271 (local address 127.0.98.3:33965)
0620 14:20:52.169516 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.169520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.169721 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:20:52.169842 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.169845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.169902 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.169958 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:52.170218 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.170929 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.172750 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.173366 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.173478 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.173543 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.173869 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.173943 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:20:52.174077 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.169350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53107 (local address 127.0.98.1:32929)
0620 14:20:52.169492 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.169497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.169513 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:20:52.169594 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.169597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.169659 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.169759 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:52.170087 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.170916 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.172780 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.173372 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.173478 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.173543 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.173870 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.173943 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:20:52.174113 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.169215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32929 (local address 127.0.98.2:53107)
0620 14:20:52.169334 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:52.169349 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:52.169459 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:52.169782 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:52.169788 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:52.169806 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:52.169924 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.169929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.171076 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.171080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.172611 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.172625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.173515 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.173519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.173741 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.173768 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:20:52.173837 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:20:52.173927 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":127}
I20260620 14:20:52.174111 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.169215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33965 (local address 127.0.98.2:35271)
0620 14:20:52.169649 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:52.169659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:52.169719 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:52.170013 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:52.170019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:52.170040 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:52.170126 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.170130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.171076 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.171080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.172611 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:52.172625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.173515 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.173519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:52.173699 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.173727 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:20:52.173816 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:20:52.173927 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":525}
I20260620 14:20:52.174892 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2350a935b41465bae6291def9f26346" candidate_uuid: "96f5529d6fb143cdae32db52d94cc28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" is_pre_election: true
I20260620 14:20:52.175220 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2350a935b41465bae6291def9f26346" candidate_uuid: "96f5529d6fb143cdae32db52d94cc28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f925011b90ff4a7e928050da17e12b4a" is_pre_election: true
I20260620 14:20:52.175283 7418 raft_consensus.cc:2454] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f5529d6fb143cdae32db52d94cc28d in term 0.
I20260620 14:20:52.175522 7691 raft_consensus.cc:2454] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f5529d6fb143cdae32db52d94cc28d in term 0.
I20260620 14:20:52.175758 7491 leader_election.cc:302] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f5529d6fb143cdae32db52d94cc28d, e15ef7c8dfbf4028bd99881804e1bc13; no voters:
I20260620 14:20:52.176054 7776 raft_consensus.cc:2781] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:52.176136 7776 raft_consensus.cc:482] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:52.176187 7776 raft_consensus.cc:3037] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:52.177209 7776 raft_consensus.cc:504] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:52.177546 7776 leader_election.cc:288] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [CANDIDATE]: Term 1 election: Requested vote from peers f925011b90ff4a7e928050da17e12b4a (127.0.98.3:33965), e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929)
I20260620 14:20:52.177875 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2350a935b41465bae6291def9f26346" candidate_uuid: "96f5529d6fb143cdae32db52d94cc28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f925011b90ff4a7e928050da17e12b4a"
I20260620 14:20:52.177875 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2350a935b41465bae6291def9f26346" candidate_uuid: "96f5529d6fb143cdae32db52d94cc28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e15ef7c8dfbf4028bd99881804e1bc13"
I20260620 14:20:52.178002 7418 raft_consensus.cc:3037] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:52.178002 7691 raft_consensus.cc:3037] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:52.179280 7418 raft_consensus.cc:2454] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f5529d6fb143cdae32db52d94cc28d in term 1.
I20260620 14:20:52.179291 7691 raft_consensus.cc:2454] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f5529d6fb143cdae32db52d94cc28d in term 1.
I20260620 14:20:52.179589 7488 leader_election.cc:302] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f5529d6fb143cdae32db52d94cc28d, f925011b90ff4a7e928050da17e12b4a; no voters:
I20260620 14:20:52.179781 7776 raft_consensus.cc:2781] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:52.180068 7776 raft_consensus.cc:686] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 1 LEADER]: Becoming Leader. State: Replica: 96f5529d6fb143cdae32db52d94cc28d, State: Running, Role: LEADER
I20260620 14:20:52.180418 7776 consensus_queue.cc:237] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:52.184118 6502 catalog_manager.cc:5495] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d reported cstate change: term changed from 0 to 1, leader changed from <none> to 96f5529d6fb143cdae32db52d94cc28d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "96f5529d6fb143cdae32db52d94cc28d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } health_report { overall_health: UNKNOWN } } }
I20260620 14:20:52.201017 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.197343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.0.98.2:35525)
0620 14:20:52.197495 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.197500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.197514 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:20:52.197660 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.197662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.197714 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.197826 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:52.198087 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.198651 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.199493 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.199717 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.199891 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.200209 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:52.200634 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:52.200766 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.200805 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.200869 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:20:52.206601 7691 raft_consensus.cc:1261] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [term 1 FOLLOWER]: Refusing update from remote peer 96f5529d6fb143cdae32db52d94cc28d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:52.206601 7418 raft_consensus.cc:1261] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [term 1 FOLLOWER]: Refusing update from remote peer 96f5529d6fb143cdae32db52d94cc28d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:52.207309 7776 consensus_queue.cc:1035] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:52.207515 7779 consensus_queue.cc:1035] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:52.218081 7783 mvcc.cc:205] Tried to move back new op lower bound from 7298929673031307264 to 7298929672933986304. Current Snapshot: MvccSnapshot[applied={T|T < 7298929673031307264}]
I20260620 14:20:52.220315 7784 mvcc.cc:205] Tried to move back new op lower bound from 7298929673031307264 to 7298929672933986304. Current Snapshot: MvccSnapshot[applied={T|T < 7298929673031307264}]
I20260620 14:20:52.225543 6502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:52.225706 6502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:52.229571 6502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71a466f760d746b2b42fe1b85085da5f: Sending DeleteTablet for 3 replicas of tablet f2350a935b41465bae6291def9f26346
I20260620 14:20:52.230432 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet f2350a935b41465bae6291def9f26346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50984
I20260620 14:20:52.230926 7791 tablet_replica.cc:330] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: stopping tablet replica
I20260620 14:20:52.231240 7791 raft_consensus.cc:2229] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:52.231273 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet f2350a935b41465bae6291def9f26346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42866
I20260620 14:20:52.231585 7791 raft_consensus.cc:2258] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:52.231802 7792 tablet_replica.cc:330] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: stopping tablet replica
I20260620 14:20:52.232137 7792 raft_consensus.cc:2229] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:52.232142 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet f2350a935b41465bae6291def9f26346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59564
I20260620 14:20:52.232496 7792 raft_consensus.cc:2258] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:52.233717 7791 ts_tablet_manager.cc:1905] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:52.233935 7792 ts_tablet_manager.cc:1905] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:52.235127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 6470
I20260620 14:20:52.237375 7792 ts_tablet_manager.cc:1918] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:52.237534 7792 log.cc:1196] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/f2350a935b41465bae6291def9f26346
I20260620 14:20:52.237586 7791 ts_tablet_manager.cc:1918] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:52.237689 7791 log.cc:1196] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/f2350a935b41465bae6291def9f26346
I20260620 14:20:52.238039 7792 ts_tablet_manager.cc:1939] T f2350a935b41465bae6291def9f26346 P e15ef7c8dfbf4028bd99881804e1bc13: Deleting consensus metadata
I20260620 14:20:52.238071 7791 ts_tablet_manager.cc:1939] T f2350a935b41465bae6291def9f26346 P f925011b90ff4a7e928050da17e12b4a: Deleting consensus metadata
I20260620 14:20:52.239495 7793 tablet_replica.cc:330] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: stopping tablet replica
I20260620 14:20:52.239894 7793 raft_consensus.cc:2229] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:52.240280 7793 raft_consensus.cc:2258] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:52.241863 7793 ts_tablet_manager.cc:1905] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:20:52.242908 7626 connection.cc:535] server connection from 127.0.0.1:50984 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 14:20:52.243332 7353 connection.cc:535] server connection from 127.0.0.1:42866 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:20:52.244964 7793 ts_tablet_manager.cc:1918] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:52.245061 7793 log.cc:1196] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/f2350a935b41465bae6291def9f26346
I20260620 14:20:52.245411 7793 ts_tablet_manager.cc:1939] T f2350a935b41465bae6291def9f26346 P 96f5529d6fb143cdae32db52d94cc28d: Deleting consensus metadata
W20260620 14:20:52.246464 7491 connection.cc:441] server connection from 127.0.0.1:59564 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59564 (request call id 1) could send its response
I20260620 14:20:52.257566 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.246937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.98.60:35659)
0620 14:20:52.247073 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.247078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.247194 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:20:52.247284 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.247287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.247338 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.247419 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:52.248897 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.249631 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.250777 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.251106 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.251172 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.253494 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.253498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.253502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.253527 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.255736 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.256275 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.256279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.256281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.256361 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.256645 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.256649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.256650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.256856 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.256955 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.257279 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.257405 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:20:52.258961 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.248399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.0.98.61:40949)
0620 14:20:52.248501 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.248507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.249141 (+ 634us) server_negotiation.cc:407] Connection header received
0620 14:20:52.249289 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.249293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.249332 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.249379 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:52.250276 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.251038 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.252782 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.253121 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.253187 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.255176 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.255180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.255182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.255211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.257026 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.257477 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.257481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.257482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.257569 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.257857 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.257860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.257862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.258073 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.258157 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.258630 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.258774 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":29,"spinlock_wait_cycles":3968}
I20260620 14:20:52.259543 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.233965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.0.98.61:40949)
0620 14:20:52.234083 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.234087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.246833 (+ 12746us) server_negotiation.cc:407] Connection header received
0620 14:20:52.247289 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.247292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.247340 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.247418 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:52.248550 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.249429 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.250738 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.251091 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.251169 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.253495 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.253498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.253502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.253527 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.256701 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.257185 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.257191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.257196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.257271 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.257581 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.257585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.257588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.257789 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.257892 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.258630 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.259354 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":571,"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":11264}
I20260620 14:20:52.267357 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:52.234061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.0.98.60:35659)
0620 14:20:52.234174 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:52.234179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:52.259027 (+ 24848us) server_negotiation.cc:407] Connection header received
0620 14:20:52.259409 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:52.259412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:52.259457 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:52.259539 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:52.260366 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.261134 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.261889 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:52.262244 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:52.262335 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:52.264013 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:52.264016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:52.264019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:52.264047 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:52.265596 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.266026 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.266030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.266031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.266116 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:52.266412 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:52.266416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:52.266418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:52.266603 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:52.266741 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:52.266980 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:20:52.267102 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:20:53.222168 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.221772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:54819)
0620 14:20:53.222005 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.222078 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
W20260620 14:20:53.222494 7465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40575 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:53.223284 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.222765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:39297)
0620 14:20:53.223068 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.223189 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
W20260620 14:20:53.223567 7602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40575 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:53.223975 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.223512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:54197)
0620 14:20:53.223825 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.223903 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":64,"threads_started":1}
W20260620 14:20:53.224200 7738 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40575 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:53.730775 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a466f760d746b2b42fe1b85085da5f)
I20260620 14:20:53.730926 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a466f760d746b2b42fe1b85085da5f)
I20260620 14:20:53.731034 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } }
I20260620 14:20:53.731177 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } }
I20260620 14:20:53.732084 7805 leader_election.cc:288] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575), 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60:35659)
I20260620 14:20:53.732134 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575), edf297fef1cf4ba788d49e4ada1c5dcc (127.0.98.61:40949)
I20260620 14:20:53.732760 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.732274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50644)
0620 14:20:53.732596 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.732670 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":88,"threads_started":1}
I20260620 14:20:53.732760 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.732234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50634)
0620 14:20:53.732596 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.732670 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":108,"threads_started":1}
W20260620 14:20:53.732936 7087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
W20260620 14:20:53.732951 7270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:53.733754 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.733523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50660)
0620 14:20:53.733631 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.733688 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:20:53.733865 7087 leader_election.cc:334] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:53.734146 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.733717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50668)
0620 14:20:53.734033 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.734085 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 14:20:53.734236 7270 leader_election.cc:334] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:53.736625 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.732401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35659 (local address 127.0.0.1:37372)
0620 14:20:53.732654 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.732671 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:53.732764 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:53.733494 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:53.733497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:53.733517 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:53.733618 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:53.733624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:53.734721 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:53.734726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:53.735523 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:53.735531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:53.736230 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:53.736234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:53.736328 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:53.736345 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:20:53.736442 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:20:53.736513 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":51,"threads_started":1}
I20260620 14:20:53.736625 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.732772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.0.98.60:35659)
0620 14:20:53.733173 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:53.733177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:53.733188 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:20:53.733228 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:53.733231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:53.733278 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:53.733370 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:53.733743 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:53.734610 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:53.735653 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:53.736117 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:53.736231 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:53.736286 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:53.736460 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:20:53.736520 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":345,"thread_start_us":58,"threads_started":1}
I20260620 14:20:53.736792 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.732627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40949 (local address 127.0.0.1:43946)
0620 14:20:53.732892 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.732904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:53.732992 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:53.733635 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:53.733639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:53.733663 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:53.733896 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:53.733905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:53.734831 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:53.734834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:53.735696 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:53.735705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:53.736353 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:53.736356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:53.736441 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:53.736465 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:20:53.736561 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:20:53.736628 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":104,"thread_start_us":48,"threads_started":1}
I20260620 14:20:53.736759 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.733092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.0.98.61:40949)
0620 14:20:53.733402 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:53.733414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:53.733426 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:20:53.733471 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:53.733476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:53.733517 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:53.733988 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:53.734081 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:53.734714 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:53.735875 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:53.736239 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:53.736352 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:53.736405 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:53.736577 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:20:53.736645 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":66,"threads_started":1}
I20260620 14:20:53.737460 7112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "820e02028d7f478ca3c196b4420e39c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" is_pre_election: true
I20260620 14:20:53.737480 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "820e02028d7f478ca3c196b4420e39c9" is_pre_election: true
I20260620 14:20:53.737649 7112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820e02028d7f478ca3c196b4420e39c9 in term 1.
I20260620 14:20:53.737644 7287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edf297fef1cf4ba788d49e4ada1c5dcc in term 1.
I20260620 14:20:53.737994 7270 leader_election.cc:302] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 820e02028d7f478ca3c196b4420e39c9, edf297fef1cf4ba788d49e4ada1c5dcc; no voters: 71a466f760d746b2b42fe1b85085da5f
I20260620 14:20:53.738001 7089 leader_election.cc:302] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 820e02028d7f478ca3c196b4420e39c9, edf297fef1cf4ba788d49e4ada1c5dcc; no voters: 71a466f760d746b2b42fe1b85085da5f
I20260620 14:20:53.738185 7806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:20:53.738246 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71a466f760d746b2b42fe1b85085da5f)
I20260620 14:20:53.738287 7806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:20:53.738334 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:20:53.738399 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71a466f760d746b2b42fe1b85085da5f)
I20260620 14:20:53.738459 7805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:20:53.739557 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } }
I20260620 14:20:53.739557 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } }
I20260620 14:20:53.739933 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [CANDIDATE]: Term 2 election: Requested vote from peers 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575), edf297fef1cf4ba788d49e4ada1c5dcc (127.0.98.61:40949)
I20260620 14:20:53.739938 7805 leader_election.cc:288] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 2 election: Requested vote from peers 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575), 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60:35659)
I20260620 14:20:53.740233 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.740029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50694)
0620 14:20:53.740123 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.740172 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:20:53.740350 7112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "820e02028d7f478ca3c196b4420e39c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc"
I20260620 14:20:53.740502 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.740029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50682)
0620 14:20:53.740382 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.740434 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20260620 14:20:53.740559 7112 raft_consensus.cc:2379] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 820e02028d7f478ca3c196b4420e39c9 in current term 2: Already voted for candidate edf297fef1cf4ba788d49e4ada1c5dcc in this term.
I20260620 14:20:53.740576 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "820e02028d7f478ca3c196b4420e39c9"
I20260620 14:20:53.740718 7287 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate edf297fef1cf4ba788d49e4ada1c5dcc in current term 2: Already voted for candidate 820e02028d7f478ca3c196b4420e39c9 in this term.
I20260620 14:20:53.741133 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.740894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50706)
0620 14:20:53.741025 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.741072 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:20:53.741248 7087 leader_election.cc:334] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:53.741318 7087 leader_election.cc:302] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edf297fef1cf4ba788d49e4ada1c5dcc; no voters: 71a466f760d746b2b42fe1b85085da5f, 820e02028d7f478ca3c196b4420e39c9
I20260620 14:20:53.741412 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:53.741210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50710)
0620 14:20:53.741292 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:53.741356 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 14:20:53.741439 7805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 14:20:53.741571 7270 leader_election.cc:334] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:53.741655 7270 leader_election.cc:302] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 820e02028d7f478ca3c196b4420e39c9; no voters: 71a466f760d746b2b42fe1b85085da5f, edf297fef1cf4ba788d49e4ada1c5dcc
I20260620 14:20:53.741780 7806 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:20:54.223448 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:54.223055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:46259)
0620 14:20:54.223303 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:54.223364 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260620 14:20:54.224547 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:54.224105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:51643)
0620 14:20:54.224382 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:54.224448 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260620 14:20:54.225139 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:54.224669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:45505)
0620 14:20:54.225023 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:54.225070 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":69,"threads_started":1}
I20260620 14:20:55.224529 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.224136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:58775)
0620 14:20:55.224372 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.224442 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260620 14:20:55.225594 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.225146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:43571)
0620 14:20:55.225412 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.225464 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260620 14:20:55.226123 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.225737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:54249)
0620 14:20:55.225979 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.226040 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":110,"threads_started":1}
I20260620 14:20:55.461910 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:55.462030 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } }
I20260620 14:20:55.462545 7821 leader_election.cc:288] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575), 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60:35659)
I20260620 14:20:55.462990 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "820e02028d7f478ca3c196b4420e39c9" is_pre_election: true
I20260620 14:20:55.463151 7287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edf297fef1cf4ba788d49e4ada1c5dcc in term 2.
I20260620 14:20:55.463236 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.462752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50988)
0620 14:20:55.463065 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.463134 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":64,"threads_started":1}
I20260620 14:20:55.463471 7089 leader_election.cc:302] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 820e02028d7f478ca3c196b4420e39c9, edf297fef1cf4ba788d49e4ada1c5dcc; no voters:
I20260620 14:20:55.463706 7821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 14:20:55.463825 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:55.463884 7821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:20:55.464255 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.464016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:50996)
0620 14:20:55.464134 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.464189 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:20:55.464377 7087 leader_election.cc:334] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:55.465088 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } }
I20260620 14:20:55.465407 7821 leader_election.cc:288] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 3 election: Requested vote from peers 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575), 820e02028d7f478ca3c196b4420e39c9 (127.0.98.60:35659)
I20260620 14:20:55.465775 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.465542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:51004)
0620 14:20:55.465649 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.465706 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:20:55.465797 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "820e02028d7f478ca3c196b4420e39c9"
I20260620 14:20:55.466058 7287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:20:55.466331 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.466165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:51018)
0620 14:20:55.466241 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.466280 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:20:55.466459 7087 leader_election.cc:334] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
I20260620 14:20:55.467156 7287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edf297fef1cf4ba788d49e4ada1c5dcc in term 3.
I20260620 14:20:55.467491 7089 leader_election.cc:302] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 820e02028d7f478ca3c196b4420e39c9, edf297fef1cf4ba788d49e4ada1c5dcc; no voters: 71a466f760d746b2b42fe1b85085da5f
I20260620 14:20:55.467706 7821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:20:55.467921 7821 raft_consensus.cc:686] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [term 3 LEADER]: Becoming Leader. State: Replica: edf297fef1cf4ba788d49e4ada1c5dcc, State: Running, Role: LEADER
I20260620 14:20:55.468189 7821 consensus_queue.cc:237] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } }
I20260620 14:20:55.469282 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader edf297fef1cf4ba788d49e4ada1c5dcc. Latest consensus state: current_term: 3 leader_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:55.469399 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:55.470610 7826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:55.472863 7826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:55.473260 7826 catalog_manager.cc:1259] Loaded cluster ID: ccd44411486a4584808cf9a7c46c2a6e
I20260620 14:20:55.473330 7826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:55.473929 7826 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:55.474311 7826 catalog_manager.cc:5879] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Loaded TSK: 0
I20260620 14:20:55.475018 7826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:20:55.553515 7102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:55.556924 7102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:55.560034 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.559752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:51044)
0620 14:20:55.559902 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.559966 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 14:20:55.560218 7287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [term 3 FOLLOWER]: Refusing update from remote peer edf297fef1cf4ba788d49e4ada1c5dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 14:20:55.560827 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.560567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.0.1:51058)
0620 14:20:55.560672 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.560741 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:20:55.560739 7824 consensus_queue.cc:1035] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 14:20:55.560949 7087 consensus_peers.cc:487] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc -> Peer 71a466f760d746b2b42fe1b85085da5f (127.0.98.62:40575): Couldn't send request to peer 71a466f760d746b2b42fe1b85085da5f. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:20:55.562165 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edf297fef1cf4ba788d49e4ada1c5dcc. Latest consensus state: current_term: 3 leader_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:55.562328 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:20:55.563429 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:55.563545 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:55.564007 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71a466f760d746b2b42fe1b85085da5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40575 } } peers { permanent_uuid: "edf297fef1cf4ba788d49e4ada1c5dcc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40949 } attrs { promote: false } } peers { permanent_uuid: "820e02028d7f478ca3c196b4420e39c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35659 } attrs { promote: false } } }
I20260620 14:20:55.564121 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e02028d7f478ca3c196b4420e39c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:20:55.573625 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.569399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.0.98.2:35525)
0620 14:20:55.569624 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:55.569629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:55.569668 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:20:55.569720 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:55.569723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:55.569769 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:55.569851 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:55.570243 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.571025 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.572376 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.572987 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.573080 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:55.573140 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:55.573424 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 14:20:55.573495 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260620 14:20:55.573624 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.568967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33965 (local address 127.0.0.1:50992)
0620 14:20:55.569079 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.569095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:55.569183 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:55.570073 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:55.570078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:55.570100 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:55.570212 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:55.570217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.571210 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.571213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:55.572179 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:55.572187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.573222 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.573225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:55.573330 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:55.573350 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:20:55.573422 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:20:55.573491 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":11}
I20260620 14:20:55.573648 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.569205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35525 (local address 127.0.0.1:59580)
0620 14:20:55.569502 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.569515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:55.569605 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:55.569980 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:55.569983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:55.569999 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:55.570103 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:55.570110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.571139 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.571143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:55.572231 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:55.572241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.573192 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.573196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:55.573279 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:55.573306 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:20:55.573392 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:20:55.573461 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":100,"threads_started":1}
I20260620 14:20:55.574014 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.569684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32929 (local address 127.0.0.1:42882)
0620 14:20:55.570052 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.570064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:55.570129 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:55.570599 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:55.570602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:55.570619 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:55.570692 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:55.570697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.571641 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.571645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:55.572750 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:55.572757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.573665 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.573670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:55.573764 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:55.573793 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:20:55.573875 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:20:55.573934 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":80,"threads_started":1}
I20260620 14:20:55.574097 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.569098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.0.98.3:33965)
0620 14:20:55.569806 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:55.569811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:55.569824 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:20:55.569888 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:55.569891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:55.569943 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:55.570308 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:55.570403 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.571046 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.572445 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.573104 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.573223 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:55.573282 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:55.573905 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 14:20:55.573978 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":62,"threads_started":1}
I20260620 14:20:55.574410 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.570000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.0.98.1:32929)
0620 14:20:55.570374 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:55.570378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:55.570389 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:20:55.570435 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:55.570438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:55.570480 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:55.570567 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:55.570811 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.571527 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.572886 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.573462 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.573554 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:55.573611 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:55.574187 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 14:20:55.574276 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":72,"threads_started":1}
I20260620 14:20:55.574579 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 1b47a0582f514306ab1a89555aca4aff (DEFAULT_TABLE table=test-table [id=152aac70fe1f4a1192c8a719b543a5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:55.574676 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 1b47a0582f514306ab1a89555aca4aff (DEFAULT_TABLE table=test-table [id=152aac70fe1f4a1192c8a719b543a5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:55.574931 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b47a0582f514306ab1a89555aca4aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:55.574939 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b47a0582f514306ab1a89555aca4aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:55.577075 7839 tablet_bootstrap.cc:492] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: Bootstrap starting.
I20260620 14:20:55.577100 7840 tablet_bootstrap.cc:492] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: Bootstrap starting.
I20260620 14:20:55.577541 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 1b47a0582f514306ab1a89555aca4aff (DEFAULT_TABLE table=test-table [id=152aac70fe1f4a1192c8a719b543a5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:55.577795 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b47a0582f514306ab1a89555aca4aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:55.578056 7839 tablet_bootstrap.cc:654] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:55.578056 7840 tablet_bootstrap.cc:654] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:55.579138 7840 tablet_bootstrap.cc:492] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: No bootstrap required, opened a new log
I20260620 14:20:55.579229 7840 ts_tablet_manager.cc:1397] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:55.579335 7839 tablet_bootstrap.cc:492] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: No bootstrap required, opened a new log
I20260620 14:20:55.579610 7839 ts_tablet_manager.cc:1397] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:20:55.579653 7840 raft_consensus.cc:348] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:55.579799 7840 raft_consensus.cc:374] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:55.579879 7840 raft_consensus.cc:729] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f925011b90ff4a7e928050da17e12b4a, State: Initialized, Role: FOLLOWER
I20260620 14:20:55.580013 7840 consensus_queue.cc:260] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:55.580075 7839 raft_consensus.cc:348] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:55.580260 7839 raft_consensus.cc:374] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:55.580319 7843 tablet_bootstrap.cc:492] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13: Bootstrap starting.
I20260620 14:20:55.580345 7839 raft_consensus.cc:729] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f5529d6fb143cdae32db52d94cc28d, State: Initialized, Role: FOLLOWER
I20260620 14:20:55.580421 7840 ts_tablet_manager.cc:1428] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:20:55.580562 7839 consensus_queue.cc:260] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:55.581071 7839 ts_tablet_manager.cc:1428] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:55.581504 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.581018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:42547)
0620 14:20:55.581383 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.581438 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297}
I20260620 14:20:55.581708 7843 tablet_bootstrap.cc:654] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:55.582202 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.581880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:46853)
0620 14:20:55.582072 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.582129 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 14:20:55.584523 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.584287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:57545)
0620 14:20:55.584408 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.584459 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:20:55.585129 7845 raft_consensus.cc:482] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:55.585240 7845 raft_consensus.cc:504] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:55.585322 7603 heartbeater.cc:503] Master 127.0.98.61:40949 was elected leader, sending a full tablet report...
I20260620 14:20:55.585837 7845 leader_election.cc:288] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f925011b90ff4a7e928050da17e12b4a (127.0.98.3:33965), e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929)
I20260620 14:20:55.585810 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b47a0582f514306ab1a89555aca4aff" candidate_uuid: "96f5529d6fb143cdae32db52d94cc28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f925011b90ff4a7e928050da17e12b4a" is_pre_election: true
I20260620 14:20:55.585949 7691 raft_consensus.cc:2454] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f5529d6fb143cdae32db52d94cc28d in term 0.
I20260620 14:20:55.586181 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b47a0582f514306ab1a89555aca4aff" candidate_uuid: "96f5529d6fb143cdae32db52d94cc28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" is_pre_election: true
I20260620 14:20:55.586251 7488 leader_election.cc:302] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f5529d6fb143cdae32db52d94cc28d, f925011b90ff4a7e928050da17e12b4a; no voters:
I20260620 14:20:55.586364 7739 heartbeater.cc:503] Master 127.0.98.61:40949 was elected leader, sending a full tablet report...
I20260620 14:20:55.586637 7845 raft_consensus.cc:2781] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:55.586719 7845 raft_consensus.cc:482] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 14:20:55.586702 7491 leader_election.cc:341] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929): Illegal state: must be running to vote when last-logged opid is not known
I20260620 14:20:55.586813 7845 raft_consensus.cc:3037] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:55.587493 7843 tablet_bootstrap.cc:492] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13: No bootstrap required, opened a new log
I20260620 14:20:55.587581 7843 ts_tablet_manager.cc:1397] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 14:20:55.588034 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.587642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:56469)
0620 14:20:55.587876 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.587935 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 14:20:55.588011 7843 raft_consensus.cc:348] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:55.588101 7843 raft_consensus.cc:374] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:55.588059 7845 raft_consensus.cc:504] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:55.588140 7843 raft_consensus.cc:729] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15ef7c8dfbf4028bd99881804e1bc13, State: Initialized, Role: FOLLOWER
I20260620 14:20:55.588253 7843 consensus_queue.cc:260] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:55.588572 7843 ts_tablet_manager.cc:1428] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:55.588622 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b47a0582f514306ab1a89555aca4aff" candidate_uuid: "96f5529d6fb143cdae32db52d94cc28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f925011b90ff4a7e928050da17e12b4a"
I20260620 14:20:55.588723 7691 raft_consensus.cc:3037] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:55.588656 7845 leader_election.cc:288] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [CANDIDATE]: Term 1 election: Requested vote from peers f925011b90ff4a7e928050da17e12b4a (127.0.98.3:33965), e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929)
I20260620 14:20:55.589200 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.588951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:48415)
0620 14:20:55.589097 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.589142 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 14:20:55.589725 7691 raft_consensus.cc:2454] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f5529d6fb143cdae32db52d94cc28d in term 1.
I20260620 14:20:55.590026 7488 leader_election.cc:302] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f5529d6fb143cdae32db52d94cc28d, f925011b90ff4a7e928050da17e12b4a; no voters:
I20260620 14:20:55.590167 7845 raft_consensus.cc:2781] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:55.590282 7845 raft_consensus.cc:686] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 1 LEADER]: Becoming Leader. State: Replica: 96f5529d6fb143cdae32db52d94cc28d, State: Running, Role: LEADER
I20260620 14:20:55.590448 7845 consensus_queue.cc:237] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } }
I20260620 14:20:55.590476 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b47a0582f514306ab1a89555aca4aff" candidate_uuid: "96f5529d6fb143cdae32db52d94cc28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e15ef7c8dfbf4028bd99881804e1bc13"
I20260620 14:20:55.590600 7418 raft_consensus.cc:3037] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:55.591657 7418 raft_consensus.cc:2454] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f5529d6fb143cdae32db52d94cc28d in term 1.
I20260620 14:20:55.592280 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.592041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:58955)
0620 14:20:55.592161 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.592214 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:20:55.592784 7102 catalog_manager.cc:5495] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d reported cstate change: term changed from 0 to 1, leader changed from <none> to 96f5529d6fb143cdae32db52d94cc28d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "96f5529d6fb143cdae32db52d94cc28d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96f5529d6fb143cdae32db52d94cc28d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 } health_report { overall_health: UNKNOWN } } }
I20260620 14:20:55.594486 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.594213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:32897)
0620 14:20:55.594346 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.594401 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:20:55.594784 7466 heartbeater.cc:503] Master 127.0.98.61:40949 was elected leader, sending a full tablet report...
I20260620 14:20:55.608960 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.606372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.0.98.2:35525)
0620 14:20:55.606476 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:55.606480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:55.606493 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:20:55.606538 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:55.606540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:55.606580 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:55.606642 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:55.606825 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.607307 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.608154 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:55.608377 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:55.608453 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:55.608639 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:55.608701 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:55.608751 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:55.608820 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:20:55.608861 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:20:55.610877 7418 raft_consensus.cc:1261] T 1b47a0582f514306ab1a89555aca4aff P e15ef7c8dfbf4028bd99881804e1bc13 [term 1 FOLLOWER]: Refusing update from remote peer 96f5529d6fb143cdae32db52d94cc28d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:55.610904 7691 raft_consensus.cc:1261] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [term 1 FOLLOWER]: Refusing update from remote peer 96f5529d6fb143cdae32db52d94cc28d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:55.611327 7845 consensus_queue.cc:1035] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15ef7c8dfbf4028bd99881804e1bc13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:55.611477 7847 consensus_queue.cc:1035] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f925011b90ff4a7e928050da17e12b4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:55.613154 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.612861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:35489)
0620 14:20:55.613027 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.613084 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 14:20:55.613721 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.613456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:44443)
0620 14:20:55.613593 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.613649 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:20:55.617753 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.617503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:47557)
0620 14:20:55.617637 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.617686 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 14:20:55.618506 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.618294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.2:40383)
0620 14:20:55.618403 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.618453 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:20:55.618837 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.616896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.3:53551)
0620 14:20:55.618729 (+ 1833us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.618777 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1767}
I20260620 14:20:55.620848 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.620454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40575 (local address 127.0.98.1:47897)
0620 14:20:55.620727 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.620778 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260620 14:20:55.620846 7100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:55.621017 7100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:55.624511 7100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edf297fef1cf4ba788d49e4ada1c5dcc: Sending DeleteTablet for 3 replicas of tablet 1b47a0582f514306ab1a89555aca4aff
I20260620 14:20:55.625175 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b47a0582f514306ab1a89555aca4aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:55 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50992
I20260620 14:20:55.625447 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b47a0582f514306ab1a89555aca4aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:55 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59580
I20260620 14:20:55.625944 7862 tablet_replica.cc:330] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: stopping tablet replica
I20260620 14:20:55.626013 7861 tablet_replica.cc:330] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: stopping tablet replica
I20260620 14:20:55.626112 7862 raft_consensus.cc:2229] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:55.626116 7861 raft_consensus.cc:2229] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:55.626147 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b47a0582f514306ab1a89555aca4aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:55 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42882
I20260620 14:20:55.626255 7861 raft_consensus.cc:2258] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:55.626317 7862 raft_consensus.cc:2258] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:55.626533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 7246
I20260620 14:20:55.627975 7862 ts_tablet_manager.cc:1905] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:55.631367 7862 ts_tablet_manager.cc:1918] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:55.631456 7862 log.cc:1196] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/1b47a0582f514306ab1a89555aca4aff
I20260620 14:20:55.631752 7862 ts_tablet_manager.cc:1939] T 1b47a0582f514306ab1a89555aca4aff P 96f5529d6fb143cdae32db52d94cc28d: Deleting consensus metadata
I20260620 14:20:55.632079 7861 ts_tablet_manager.cc:1905] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:55.632489 7087 catalog_manager.cc:4841] TS 96f5529d6fb143cdae32db52d94cc28d (127.0.98.2:35525): tablet 1b47a0582f514306ab1a89555aca4aff (table test-table [id=152aac70fe1f4a1192c8a719b543a5a5]) successfully deleted
I20260620 14:20:55.633965 7861 ts_tablet_manager.cc:1918] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:55.634126 7861 log.cc:1196] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/1b47a0582f514306ab1a89555aca4aff
I20260620 14:20:55.634447 7861 ts_tablet_manager.cc:1939] T 1b47a0582f514306ab1a89555aca4aff P f925011b90ff4a7e928050da17e12b4a: Deleting consensus metadata
I20260620 14:20:55.634680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 7472
I20260620 14:20:55.634771 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:55.634489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32929 (local address 127.0.0.1:42888)
0620 14:20:55.634622 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:55.634675 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:20:55.636404 7089 catalog_manager.cc:4584] TS e15ef7c8dfbf4028bd99881804e1bc13 (127.0.98.1:32929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b47a0582f514306ab1a89555aca4aff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32929: connect: Connection refused (error 111)
I20260620 14:20:55.636417 7086 catalog_manager.cc:4841] TS f925011b90ff4a7e928050da17e12b4a (127.0.98.3:33965): tablet 1b47a0582f514306ab1a89555aca4aff (table test-table [id=152aac70fe1f4a1192c8a719b543a5a5]) successfully deleted
I20260620 14:20:55.641868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 7608
I20260620 14:20:55.647372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 6966
I20260620 14:20:55.653280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 7074
2026-06-20T14:20:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6981 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:55 dist-test-slave-87nc krb5kdc[7866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:55 dist-test-slave-87nc krb5kdc[7866](info): set up 2 sockets
Jun 20 14:20:55 dist-test-slave-87nc krb5kdc[7866](info): commencing operation
krb5kdc: starting...
W20260620 14:20:56.688259 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:56 dist-test-slave-87nc krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:20:56.806497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35719
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:56.913786 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:56.914103 7879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:56.914184 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:56.918565 7879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:56.918668 7879 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:20:56.918718 7879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:56.918784 7879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:56.918823 7879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:56.918859 7879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:56.924067 7879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35719
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:56.925307 7879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:56.926396 7879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:56.935039 7885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:56.934964 7887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:56.934964 7884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:56.935065 7879 server_base.cc:1034] running on GCE node
I20260620 14:20:56.935873 7879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:56.937032 7879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:56.938225 7879 hybrid_clock.cc:648] HybridClock initialized: now 1781965256938213 us; error 35 us; skew 500 ppm
Jun 20 14:20:56 dist-test-slave-87nc krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965256, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:56.941068 7879 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:20:56.942045 7879 webserver.cc:460] Webserver started at http://127.0.98.62:37519/ using document root <none> and password file <none>
I20260620 14:20:56.942621 7879 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:56.942714 7879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:56.942955 7879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:56.945047 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "f1820151c77a409c996d34e4af7b4232"
format_stamp: "Formatted at 2026-06-20 14:20:56 on dist-test-slave-87nc"
server_key: "3e7c2306f7f55581c70c8624e20154db"
server_key_iv: "5786d1181f1156d95ab944f332c4e742"
server_key_version: "encryptionkey@0"
I20260620 14:20:56.945621 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1820151c77a409c996d34e4af7b4232"
format_stamp: "Formatted at 2026-06-20 14:20:56 on dist-test-slave-87nc"
server_key: "3e7c2306f7f55581c70c8624e20154db"
server_key_iv: "5786d1181f1156d95ab944f332c4e742"
server_key_version: "encryptionkey@0"
I20260620 14:20:56.949342 7879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:20:56.951747 7894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:56.953001 7879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:20:56.953150 7879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "f1820151c77a409c996d34e4af7b4232"
format_stamp: "Formatted at 2026-06-20 14:20:56 on dist-test-slave-87nc"
server_key: "3e7c2306f7f55581c70c8624e20154db"
server_key_iv: "5786d1181f1156d95ab944f332c4e742"
server_key_version: "encryptionkey@0"
I20260620 14:20:56.953264 7879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:56.969233 7879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:56.972056 7879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:56.972245 7879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:56.979862 7879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35719
I20260620 14:20:56.979846 7956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35719 every 8 connection(s)
I20260620 14:20:56.980789 7879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:56.982225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 7879
I20260620 14:20:56.982374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:56.982653 392 external_mini_cluster.cc:1428] Setting key 1456092cdddf7fabed26ac0ec82b7ef1
I20260620 14:20:56.983628 7957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:56.989933 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: Bootstrap starting.
Jun 20 14:20:56 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:56.992579 7957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:56.993548 7957 log.cc:824] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:56.995719 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: No bootstrap required, opened a new log
I20260620 14:20:56.997455 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:56.984070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.0.98.62:35719)
0620 14:20:56.984473 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:56.984488 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:56.984522 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:20:56.985154 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:56.985169 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:56.985503 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:56.985778 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:56.986663 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:56.987606 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:56.988785 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:56.989227 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:56.989349 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:56.992065 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:56.992071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:56.992086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:56.992115 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:56.994586 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:56.995006 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:56.995011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:56.995017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:56.995139 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:56.995400 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:56.995403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:56.995404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:56.995900 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:56.996083 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:56.996284 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 14:20:56.996566 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":136,"threads_started":1}
I20260620 14:20:56.999277 7957 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1820151c77a409c996d34e4af7b4232" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35719 } }
I20260620 14:20:56.999567 7957 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:56.999630 7957 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1820151c77a409c996d34e4af7b4232, State: Initialized, Role: FOLLOWER
I20260620 14:20:57.000108 7957 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1820151c77a409c996d34e4af7b4232" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35719 } }
I20260620 14:20:57.000254 7957 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:57.000316 7957 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:57.000406 7957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:57.001236 7957 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1820151c77a409c996d34e4af7b4232" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35719 } }
I20260620 14:20:57.001552 7957 leader_election.cc:302] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1820151c77a409c996d34e4af7b4232; no voters:
I20260620 14:20:57.001848 7957 leader_election.cc:288] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:57.001942 7962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:57.002198 7962 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [term 1 LEADER]: Becoming Leader. State: Replica: f1820151c77a409c996d34e4af7b4232, State: Running, Role: LEADER
I20260620 14:20:57.002576 7962 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1820151c77a409c996d34e4af7b4232" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35719 } }
I20260620 14:20:57.002990 7957 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:57.004153 7964 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1820151c77a409c996d34e4af7b4232. Latest consensus state: current_term: 1 leader_uuid: "f1820151c77a409c996d34e4af7b4232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1820151c77a409c996d34e4af7b4232" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35719 } } }
I20260620 14:20:57.004273 7964 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:57.004602 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1820151c77a409c996d34e4af7b4232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1820151c77a409c996d34e4af7b4232" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35719 } } }
I20260620 14:20:57.004750 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232 [sys.catalog]: This master's current role is: LEADER
W20260620 14:20:57.007552 7977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:20:57.007642 7977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:20:57.007719 7971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:57.008554 7971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:57.013617 7971 catalog_manager.cc:1347] Generated new cluster ID: fe9cc8c15f8248b3bda34ff76ac7501e
I20260620 14:20:57.013679 7971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:57.035218 7971 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:57.036598 7971 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:57.051308 7971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: Generated new TSK 0
I20260620 14:20:57.052198 7971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:20:57.082685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35719
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:57.191001 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:57.191249 7983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:57.191309 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:57.195346 7983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:57.195407 7983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:57.195446 7983 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:20:57.195519 7983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:57.200644 7983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:57.202073 7983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:57.203312 7983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:57.213536 7988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:57.213606 7989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:57.213596 7991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:57.213902 7983 server_base.cc:1034] running on GCE node
I20260620 14:20:57.214310 7983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:57.214973 7983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:57.216154 7983 hybrid_clock.cc:648] HybridClock initialized: now 1781965257216112 us; error 40 us; skew 500 ppm
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:57.218936 7983 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:20:57.219892 7983 webserver.cc:460] Webserver started at http://127.0.98.1:41031/ using document root <none> and password file <none>
I20260620 14:20:57.220446 7983 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:57.220510 7983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:57.220739 7983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:57.222787 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "74cd9547ed204da6aadcbba6ee230dd6"
format_stamp: "Formatted at 2026-06-20 14:20:57 on dist-test-slave-87nc"
server_key: "e1be705f316f902698f99059016a0d3f"
server_key_iv: "c52d41458b4ffb0ef152802a418832d5"
server_key_version: "encryptionkey@0"
I20260620 14:20:57.223384 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74cd9547ed204da6aadcbba6ee230dd6"
format_stamp: "Formatted at 2026-06-20 14:20:57 on dist-test-slave-87nc"
server_key: "e1be705f316f902698f99059016a0d3f"
server_key_iv: "c52d41458b4ffb0ef152802a418832d5"
server_key_version: "encryptionkey@0"
I20260620 14:20:57.227130 7983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:20:57.229564 7998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:57.230710 7983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:20:57.230849 7983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "74cd9547ed204da6aadcbba6ee230dd6"
format_stamp: "Formatted at 2026-06-20 14:20:57 on dist-test-slave-87nc"
server_key: "e1be705f316f902698f99059016a0d3f"
server_key_iv: "c52d41458b4ffb0ef152802a418832d5"
server_key_version: "encryptionkey@0"
I20260620 14:20:57.230962 7983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:57.256114 7983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:57.259222 7983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:57.259438 7983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:57.260993 7983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:57.261068 7983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:57.261134 7983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:57.261178 7983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:57.272044 7983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46779
I20260620 14:20:57.272133 8113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46779 every 8 connection(s)
I20260620 14:20:57.272930 7983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:57.274387 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.261829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33355 (local address 127.0.98.62:35719)
0620 14:20:57.261999 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.262006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.262797 (+ 791us) server_negotiation.cc:407] Connection header received
0620 14:20:57.263948 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.263953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.264019 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.264124 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:57.265488 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.266024 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.267173 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.267573 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.267689 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.270572 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:57.270577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:57.270580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:57.270607 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:57.272340 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.272832 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.272836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.272838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.272920 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.273240 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.273247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.273248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.273483 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:57.273578 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.274132 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.274255 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:20:57.275142 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.262188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35719 (local address 127.0.98.1:33355)
0620 14:20:57.262625 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.262677 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.263689 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.264323 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.264329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.264766 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:57.265350 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.265373 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.266153 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.266157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.267049 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.267057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.267730 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.267734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.267936 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.268542 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:57.268569 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:57.270400 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:57.272446 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.272452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.272465 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.272712 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.273008 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.273013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.273015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.273150 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.273629 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:57.273634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:57.273905 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.274063 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.274322 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":140,"threads_started":1}
I20260620 14:20:57.276428 8114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35719
I20260620 14:20:57.276571 8114 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:57.277053 8114 heartbeater.cc:511] Master 127.0.98.62:35719 requested a full tablet report, sending...
I20260620 14:20:57.278541 7910 ts_manager.cc:194] Registered new tserver with Master: 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779)
I20260620 14:20:57.279057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 7983
I20260620 14:20:57.279178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:57.279448 392 external_mini_cluster.cc:1428] Setting key cb945a751b45ba0cb2d3ba732b402715
I20260620 14:20:57.280023 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33355
I20260620 14:20:57.286031 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.278557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43215 (local address 127.0.98.62:35719)
0620 14:20:57.278680 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.278683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.278832 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:20:57.278923 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.278929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.278981 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.279067 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:57.279550 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.280220 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.281306 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.281690 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.281812 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.283083 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:57.283086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:57.283088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:57.283108 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:57.284472 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.284942 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.284945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.284946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.285012 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.285289 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.285291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.285292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.285433 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:57.285510 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.285836 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.285921 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:20:57.286217 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.278438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35719 (local address 127.0.98.1:43215)
0620 14:20:57.278734 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.278748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.278845 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.279096 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.279099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.279302 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:57.279425 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.279432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.280340 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.280343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.281179 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.281188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.281818 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.281821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.281913 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.282459 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:57.282490 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:57.282948 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:57.284581 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.284587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.284593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.284852 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.285102 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.285107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.285110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.285202 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.285526 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:57.285549 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:57.285663 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.285785 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.285957 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:20:57.304527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35719
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:57.413549 8123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:57.413805 8123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:57.413889 8123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:57.417883 8123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:57.417937 8123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:57.417975 8123 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:20:57.418006 8123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:57.423233 8123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:57.424556 8123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:57.425668 8123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:57.435400 8129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:57.435451 8128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:57.435420 8131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:57.435576 8123 server_base.cc:1034] running on GCE node
I20260620 14:20:57.436272 8123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:57.436892 8123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:57.438087 8123 hybrid_clock.cc:648] HybridClock initialized: now 1781965257438070 us; error 28 us; skew 500 ppm
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:57.440905 8123 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:20:57.441758 8123 webserver.cc:460] Webserver started at http://127.0.98.2:43311/ using document root <none> and password file <none>
I20260620 14:20:57.442329 8123 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:57.442420 8123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:57.442643 8123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:57.444665 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "0c9ffa7559154e4a8c98c1cc3db82795"
format_stamp: "Formatted at 2026-06-20 14:20:57 on dist-test-slave-87nc"
server_key: "0f6fd8ac20e291ff55f5545bf0c1cf3c"
server_key_iv: "f401a6c3ef66ff1ebcfcca27434ce810"
server_key_version: "encryptionkey@0"
I20260620 14:20:57.445199 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0c9ffa7559154e4a8c98c1cc3db82795"
format_stamp: "Formatted at 2026-06-20 14:20:57 on dist-test-slave-87nc"
server_key: "0f6fd8ac20e291ff55f5545bf0c1cf3c"
server_key_iv: "f401a6c3ef66ff1ebcfcca27434ce810"
server_key_version: "encryptionkey@0"
I20260620 14:20:57.448808 8123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:20:57.451161 8138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:57.452489 8123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:57.452613 8123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "0c9ffa7559154e4a8c98c1cc3db82795"
format_stamp: "Formatted at 2026-06-20 14:20:57 on dist-test-slave-87nc"
server_key: "0f6fd8ac20e291ff55f5545bf0c1cf3c"
server_key_iv: "f401a6c3ef66ff1ebcfcca27434ce810"
server_key_version: "encryptionkey@0"
I20260620 14:20:57.452719 8123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:57.469426 8123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:57.472456 8123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:57.472661 8123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:57.474077 8123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:57.474151 8123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:57.474218 8123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:57.474301 8123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:57.485042 8123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45211
I20260620 14:20:57.485091 8253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45211 every 8 connection(s)
I20260620 14:20:57.486181 8123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:57.487749 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.475003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38733 (local address 127.0.98.62:35719)
0620 14:20:57.475137 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.475142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.476020 (+ 878us) server_negotiation.cc:407] Connection header received
0620 14:20:57.477058 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.477061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.477117 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.477175 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:57.478920 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.479842 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.481094 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.481318 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.481433 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.484009 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:57.484013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:57.484016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:57.484046 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:57.485598 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.486066 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.486072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.486076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.486184 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.486549 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.486554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.486558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.486772 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:57.486908 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.487467 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.487585 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:20:57.488519 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.475402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35719 (local address 127.0.98.2:38733)
0620 14:20:57.475865 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.475919 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.476807 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.477586 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.477611 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.478060 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:57.478751 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.478771 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.479967 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.479971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.480929 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.480961 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.481480 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.481484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.481675 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.482296 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:57.482319 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:57.483854 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:57.485718 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.485721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.485732 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.485959 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.486288 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.486292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.486297 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.486432 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.486906 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:57.486913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:57.487192 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.487414 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.487710 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":134,"threads_started":1}
I20260620 14:20:57.489848 8254 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35719
I20260620 14:20:57.489996 8254 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:57.490516 8254 heartbeater.cc:511] Master 127.0.98.62:35719 requested a full tablet report, sending...
I20260620 14:20:57.490538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 8123
I20260620 14:20:57.490677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:57.491008 392 external_mini_cluster.cc:1428] Setting key 2545f2860ac8bbd57fdf7e71daebe516
I20260620 14:20:57.491544 7911 ts_manager.cc:194] Registered new tserver with Master: 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211)
I20260620 14:20:57.492394 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38733
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:20:57.500432 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.491929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35719 (local address 127.0.98.2:54871)
0620 14:20:57.492228 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.492242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.492369 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.492648 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.492654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.492884 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:57.494374 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.494387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.495135 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.495138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.495709 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.495717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.496249 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.496253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.496342 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.496928 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:57.496942 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:57.497248 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:57.498891 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.498895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.498898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.499096 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.499378 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.499382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.499384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.499470 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.499939 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:57.499943 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:57.500039 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.500154 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.500280 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
I20260620 14:20:57.500703 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.492120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54871 (local address 127.0.98.62:35719)
0620 14:20:57.492221 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.492225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.492304 (+ 79us) server_negotiation.cc:407] Connection header received
0620 14:20:57.492479 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.492482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.492526 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.492606 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:57.494515 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.495019 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.495907 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.496129 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.496246 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.497386 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:57.497390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:57.497393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:57.497415 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:57.498773 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.499195 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.499200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.499202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.499284 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.499560 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.499564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.499566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.499828 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:57.499939 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.500269 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.500411 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:20:57.518265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35719
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:57.626750 8263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:57.627071 8263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:57.627151 8263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:57.631145 8263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:57.631239 8263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:57.631316 8263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:20:57.631414 8263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:57.636821 8263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:57.637841 8263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:57.638833 8263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:20:57.648264 8263 server_base.cc:1034] running on GCE node
W20260620 14:20:57.648228 8271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:57.648294 8268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:57.648360 8269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:57.648870 8263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:57.649462 8263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:57.650615 8263 hybrid_clock.cc:648] HybridClock initialized: now 1781965257650595 us; error 32 us; skew 500 ppm
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:57.653244 8263 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:20:57.654067 8263 webserver.cc:460] Webserver started at http://127.0.98.3:33241/ using document root <none> and password file <none>
I20260620 14:20:57.654567 8263 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:57.654631 8263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:57.654786 8263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:20:57.656855 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d72c244a5a44e6db348fad33b71a6e6"
format_stamp: "Formatted at 2026-06-20 14:20:57 on dist-test-slave-87nc"
server_key: "f7ac03b4a967d9ea91caf0843a165dd1"
server_key_iv: "3e329a46876ffaa2bb99511b0e6bbb17"
server_key_version: "encryptionkey@0"
I20260620 14:20:57.657399 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d72c244a5a44e6db348fad33b71a6e6"
format_stamp: "Formatted at 2026-06-20 14:20:57 on dist-test-slave-87nc"
server_key: "f7ac03b4a967d9ea91caf0843a165dd1"
server_key_iv: "3e329a46876ffaa2bb99511b0e6bbb17"
server_key_version: "encryptionkey@0"
I20260620 14:20:57.660862 8263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:20:57.663120 8278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:57.664384 8263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:57.664520 8263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "6d72c244a5a44e6db348fad33b71a6e6"
format_stamp: "Formatted at 2026-06-20 14:20:57 on dist-test-slave-87nc"
server_key: "f7ac03b4a967d9ea91caf0843a165dd1"
server_key_iv: "3e329a46876ffaa2bb99511b0e6bbb17"
server_key_version: "encryptionkey@0"
I20260620 14:20:57.664629 8263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:57.677768 8263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:57.680507 8263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:57.680698 8263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:57.682147 8263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:57.682195 8263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:57.682233 8263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:57.682288 8263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:57.693521 8263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36513
I20260620 14:20:57.693594 8393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36513 every 8 connection(s)
I20260620 14:20:57.694593 8263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:57.696146 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.682926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42729 (local address 127.0.98.62:35719)
0620 14:20:57.683093 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.683099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.683904 (+ 805us) server_negotiation.cc:407] Connection header received
0620 14:20:57.684868 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.684871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.684929 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.685014 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:57.686445 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.687057 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.688387 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.688674 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.688796 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.692155 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:57.692162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:57.692167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:57.692195 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:57.694003 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.694499 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.694506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.694510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.694597 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.694954 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.694957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.694959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.695197 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:57.695299 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.695933 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.696030 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:20:57.696951 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.683284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35719 (local address 127.0.98.3:42729)
0620 14:20:57.683725 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.683796 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.684655 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.685328 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.685339 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.685774 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:57.686296 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.686311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.687191 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.687199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.688262 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.688271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.688849 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.688854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.689009 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.689783 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:57.689802 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:57.691942 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:57.694108 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.694113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.694123 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.694365 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.694698 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.694701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.694703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.694850 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.695386 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:57.695392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:57.695659 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.695874 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.696120 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":137,"threads_started":1}
I20260620 14:20:57.698192 8394 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35719
I20260620 14:20:57.698351 8394 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:57.698823 8394 heartbeater.cc:511] Master 127.0.98.62:35719 requested a full tablet report, sending...
I20260620 14:20:57.699827 7911 ts_manager.cc:194] Registered new tserver with Master: 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513)
I20260620 14:20:57.700414 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42729
I20260620 14:20:57.704162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 8263
I20260620 14:20:57.704316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:57.704648 392 external_mini_cluster.cc:1428] Setting key dd86299e834df3c0bbe0daae103c77fb
I20260620 14:20:57.706875 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:20:57.707583 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.700365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33221 (local address 127.0.98.62:35719)
0620 14:20:57.700494 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.700499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.700676 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:20:57.700780 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.700783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.700842 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.700924 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:57.701396 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.701877 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.702762 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.702963 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.703037 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.704191 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:57.704194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:57.704197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:57.704219 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:57.705857 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.706321 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.706324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.706325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.706412 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.706740 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.706744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.706746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.707020 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:57.707127 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.707368 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.707462 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:20:57.707705 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.700305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35719 (local address 127.0.98.3:33221)
0620 14:20:57.700586 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.700600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.700701 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.700944 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.700946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.701160 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:57.701276 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.701296 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.702001 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.702005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.702668 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.702676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.703066 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.703068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.703130 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.703700 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:57.703711 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:57.704067 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:57.705959 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.705963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.705965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.706198 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.706564 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.706566 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.706568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.706643 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.707129 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:57.707132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:57.707222 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.707334 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.707487 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:20:57.730785 7960 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:57.724556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.0.98.62:35719)
0620 14:20:57.724788 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.724793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.724807 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:20:57.724854 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.724857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.724902 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.724986 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:57.725706 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.726211 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.726998 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.727201 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.727274 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.730605 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43626: BlockingRecv error: recv got EOF from 127.0.0.1:43626 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:57.739686 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.732184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.0.98.62:35719)
0620 14:20:57.732299 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.732302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.732500 (+ 198us) server_negotiation.cc:407] Connection header received
0620 14:20:57.732589 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.732591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.732626 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.732693 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:57.733210 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.733687 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.734453 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.734706 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.734782 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.736957 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:57.736960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:57.736962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:57.736983 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:57.738410 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.738774 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.738776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.738777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.738854 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.739060 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.739062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.739063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.739219 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:57.739316 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.739486 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.739574 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:20:57.742621 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:57.745677 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:57.759610 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.753383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45211 (local address 127.0.0.1:40570)
0620 14:20:57.753694 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.753739 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.753862 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.754998 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.755004 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.755026 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:57.755178 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.755188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.756503 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.756509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.758302 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.758312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.759085 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.759088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.759193 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.759229 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.759334 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.759442 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":88,"thread_start_us":92,"threads_started":1}
I20260620 14:20:57.759814 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.753480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.0.98.2:45211)
0620 14:20:57.754491 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.754504 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.754527 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:20:57.754623 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.754631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.754835 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.755002 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:57.755302 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.756404 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.758437 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.758966 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.759088 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.759249 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.759533 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.759628 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":90,"threads_started":1}
I20260620 14:20:57.760109 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.754139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.0.98.1:46779)
0620 14:20:57.754767 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.754780 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.754801 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:20:57.754870 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.754874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.755074 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.755228 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:57.755489 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.756772 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.758416 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.759170 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.759310 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.759449 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.759676 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.759948 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":90,"threads_started":1}
I20260620 14:20:57.760182 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.754074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36513 (local address 127.0.0.1:43330)
0620 14:20:57.754305 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.754317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.754381 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.755459 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.755462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.755479 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:57.755580 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.755588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.757295 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.757299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.758219 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.758228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.759502 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.759505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.759598 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.759619 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.759690 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.759752 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260620 14:20:57.759864 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.753473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46779 (local address 127.0.0.1:50294)
0620 14:20:57.754246 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.754259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.754356 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.755217 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.755220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.755237 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:57.755354 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.755361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.756893 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.756896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.758302 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.758312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.759326 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.759331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.759418 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.759436 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.759517 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.759566 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":102,"thread_start_us":97,"threads_started":1}
I20260620 14:20:57.760845 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.754571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.98.3:36513)
0620 14:20:57.755010 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.755021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.755042 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:20:57.755109 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.755116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.755317 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.755476 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:57.755667 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.756803 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.758365 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.759167 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.759285 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.759439 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.759948 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.760062 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":100,"threads_started":1}
I20260620 14:20:57.762614 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 1f81ff7cb373489b824ad9d99d5fe41c (DEFAULT_TABLE table=test-table [id=7f82ffe37ca643648a4566c2984de97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:57.762645 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 1f81ff7cb373489b824ad9d99d5fe41c (DEFAULT_TABLE table=test-table [id=7f82ffe37ca643648a4566c2984de97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:57.763526 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f81ff7cb373489b824ad9d99d5fe41c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:57.763600 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f81ff7cb373489b824ad9d99d5fe41c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:57.765447 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 1f81ff7cb373489b824ad9d99d5fe41c (DEFAULT_TABLE table=test-table [id=7f82ffe37ca643648a4566c2984de97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:57.766360 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f81ff7cb373489b824ad9d99d5fe41c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:57.770237 8422 tablet_bootstrap.cc:492] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: Bootstrap starting.
I20260620 14:20:57.772346 8422 tablet_bootstrap.cc:654] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:57.773018 8422 log.cc:824] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:57.773173 8424 tablet_bootstrap.cc:492] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: Bootstrap starting.
I20260620 14:20:57.773715 8423 tablet_bootstrap.cc:492] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: Bootstrap starting.
I20260620 14:20:57.775020 8422 tablet_bootstrap.cc:492] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: No bootstrap required, opened a new log
I20260620 14:20:57.775206 8422 ts_tablet_manager.cc:1397] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:20:57.775332 8424 tablet_bootstrap.cc:654] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:57.775687 8423 tablet_bootstrap.cc:654] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:57.776171 8424 log.cc:824] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:57.776468 8423 log.cc:824] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:57.778048 8424 tablet_bootstrap.cc:492] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: No bootstrap required, opened a new log
I20260620 14:20:57.778244 8424 ts_tablet_manager.cc:1397] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:20:57.778329 8423 tablet_bootstrap.cc:492] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: No bootstrap required, opened a new log
I20260620 14:20:57.778506 8423 ts_tablet_manager.cc:1397] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:20:57.779256 8422 raft_consensus.cc:348] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.779525 8422 raft_consensus.cc:374] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:57.779609 8422 raft_consensus.cc:729] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9ffa7559154e4a8c98c1cc3db82795, State: Initialized, Role: FOLLOWER
I20260620 14:20:57.780151 8422 consensus_queue.cc:260] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.780804 8254 heartbeater.cc:503] Master 127.0.98.62:35719 was elected leader, sending a full tablet report...
I20260620 14:20:57.781134 8422 ts_tablet_manager.cc:1428] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 14:20:57.781337 8428 raft_consensus.cc:482] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:57.781447 8428 raft_consensus.cc:504] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.781749 8424 raft_consensus.cc:348] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.782018 8424 raft_consensus.cc:374] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:57.782096 8424 raft_consensus.cc:729] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d72c244a5a44e6db348fad33b71a6e6, State: Initialized, Role: FOLLOWER
I20260620 14:20:57.782229 8423 raft_consensus.cc:348] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.782526 8423 raft_consensus.cc:374] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:57.782614 8423 raft_consensus.cc:729] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cd9547ed204da6aadcbba6ee230dd6, State: Initialized, Role: FOLLOWER
I20260620 14:20:57.783062 8423 consensus_queue.cc:260] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.783635 8424 consensus_queue.cc:260] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.784108 8423 ts_tablet_manager.cc:1428] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:20:57.784765 8424 ts_tablet_manager.cc:1428] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:20:57.785079 8394 heartbeater.cc:503] Master 127.0.98.62:35719 was elected leader, sending a full tablet report...
I20260620 14:20:57.785257 8114 heartbeater.cc:503] Master 127.0.98.62:35719 was elected leader, sending a full tablet report...
I20260620 14:20:57.786988 8428 leader_election.cc:288] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779), 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513)
I20260620 14:20:57.787752 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.783166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46779 (local address 127.0.98.2:53923)
0620 14:20:57.783404 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.783419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.783502 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.784102 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.784105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.784125 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:57.784246 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.784253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.785149 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.785154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.786310 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.786319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.787338 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.787342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.787447 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.787472 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.787550 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.787633 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 14:20:57.788203 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.783656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53923 (local address 127.0.98.1:46779)
0620 14:20:57.783760 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.783806 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.783821 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:57.783865 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.783868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.783919 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.783990 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:57.784370 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.785039 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.786452 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.787042 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.787149 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.787214 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.788015 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.788089 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:20:57.788619 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f81ff7cb373489b824ad9d99d5fe41c" candidate_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cd9547ed204da6aadcbba6ee230dd6" is_pre_election: true
I20260620 14:20:57.788877 8068 raft_consensus.cc:2454] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c9ffa7559154e4a8c98c1cc3db82795 in term 0.
I20260620 14:20:57.789471 8141 leader_election.cc:302] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9ffa7559154e4a8c98c1cc3db82795, 74cd9547ed204da6aadcbba6ee230dd6; no voters:
I20260620 14:20:57.789747 8428 raft_consensus.cc:2781] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:57.789841 8428 raft_consensus.cc:482] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:57.789883 8428 raft_consensus.cc:3037] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:57.791100 8428 raft_consensus.cc:504] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.791401 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.786372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36513 (local address 127.0.98.2:42023)
0620 14:20:57.787244 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.787258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.787379 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.787686 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.787689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.787705 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:57.787831 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.787838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.788842 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.788846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.789937 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.789948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.790952 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.790961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.791071 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.791100 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.791185 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.791250 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":93,"threads_started":1}
I20260620 14:20:57.791402 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.786750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42023 (local address 127.0.98.3:36513)
0620 14:20:57.786858 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.786862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.787346 (+ 484us) server_negotiation.cc:407] Connection header received
0620 14:20:57.787459 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.787462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.787510 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.787579 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:57.787975 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.788705 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.790043 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.790426 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.790509 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.790552 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.791200 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.791295 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:20:57.791800 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f81ff7cb373489b824ad9d99d5fe41c" candidate_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cd9547ed204da6aadcbba6ee230dd6"
I20260620 14:20:57.791916 8068 raft_consensus.cc:3037] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:57.791496 8428 leader_election.cc:288] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [CANDIDATE]: Term 1 election: Requested vote from peers 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779), 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513)
I20260620 14:20:57.792214 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f81ff7cb373489b824ad9d99d5fe41c" candidate_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d72c244a5a44e6db348fad33b71a6e6"
I20260620 14:20:57.792225 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f81ff7cb373489b824ad9d99d5fe41c" candidate_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d72c244a5a44e6db348fad33b71a6e6" is_pre_election: true
I20260620 14:20:57.792491 8348 raft_consensus.cc:2454] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c9ffa7559154e4a8c98c1cc3db82795 in term 0.
I20260620 14:20:57.793148 8068 raft_consensus.cc:2454] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9ffa7559154e4a8c98c1cc3db82795 in term 1.
I20260620 14:20:57.793470 8141 leader_election.cc:302] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c9ffa7559154e4a8c98c1cc3db82795, 74cd9547ed204da6aadcbba6ee230dd6; no voters: 6d72c244a5a44e6db348fad33b71a6e6
I20260620 14:20:57.793663 8428 raft_consensus.cc:2781] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:57.793761 8428 raft_consensus.cc:686] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 LEADER]: Becoming Leader. State: Replica: 0c9ffa7559154e4a8c98c1cc3db82795, State: Running, Role: LEADER
I20260620 14:20:57.794051 8428 consensus_queue.cc:237] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.796798 7910 catalog_manager.cc:5495] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } health_report { overall_health: UNKNOWN } } }
I20260620 14:20:57.811857 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.808346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.0.98.2:45211)
0620 14:20:57.808500 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.808505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.808520 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:57.808643 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.808645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.808699 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.808810 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:57.809044 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.809548 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.810394 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.810619 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.810726 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.811008 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:57.811436 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:57.811560 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.811623 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.811676 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:20:57.817157 8347 raft_consensus.cc:3037] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:57.817327 8068 raft_consensus.cc:1261] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Refusing update from remote peer 0c9ffa7559154e4a8c98c1cc3db82795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:57.817967 8429 consensus_queue.cc:1035] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:57.818812 8347 raft_consensus.cc:1261] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Refusing update from remote peer 0c9ffa7559154e4a8c98c1cc3db82795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:57.819330 8429 consensus_queue.cc:1035] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:57.826591 8438 mvcc.cc:205] Tried to move back new op lower bound from 7298929696012906496 to 7298929695927144448. Current Snapshot: MvccSnapshot[applied={T|T < 7298929696012906496}]
I20260620 14:20:57.826633 8436 mvcc.cc:205] Tried to move back new op lower bound from 7298929696012906496 to 7298929695927144448. Current Snapshot: MvccSnapshot[applied={T|T < 7298929696012906496}]
I20260620 14:20:57.834837 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:57.835011 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:57.837373 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: Sending DeleteTablet for 3 replicas of tablet 1f81ff7cb373489b824ad9d99d5fe41c
I20260620 14:20:57.838065 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f81ff7cb373489b824ad9d99d5fe41c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50294
I20260620 14:20:57.838259 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f81ff7cb373489b824ad9d99d5fe41c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43330
I20260620 14:20:57.838704 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f81ff7cb373489b824ad9d99d5fe41c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40570
I20260620 14:20:57.838759 8446 tablet_replica.cc:330] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: stopping tablet replica
I20260620 14:20:57.838822 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:20:57.839048 8446 raft_consensus.cc:2229] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 14:20:57.839162 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:57.839382 8446 raft_consensus.cc:2258] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:57.839406 8447 tablet_replica.cc:330] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: stopping tablet replica
I20260620 14:20:57.839830 8447 raft_consensus.cc:2229] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:57.840267 8447 raft_consensus.cc:2258] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:57.841128 8446 ts_tablet_manager.cc:1905] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:57.841871 8447 ts_tablet_manager.cc:1905] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:57.843847 8446 ts_tablet_manager.cc:1918] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:57.844117 8445 tablet_replica.cc:330] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: stopping tablet replica
I20260620 14:20:57.844873 8188 tablet_service.cc:1467] Processing CreateTablet for tablet e755baf9358248df98673ed087d7e8fd (DEFAULT_TABLE table=test-table [id=2826e07dc2bf42c5a1ba94feb3f543f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:57.845124 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e755baf9358248df98673ed087d7e8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:57.845121 8445 raft_consensus.cc:2229] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:57.845419 8445 raft_consensus.cc:2258] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:57.844719 8328 tablet_service.cc:1467] Processing CreateTablet for tablet e755baf9358248df98673ed087d7e8fd (DEFAULT_TABLE table=test-table [id=2826e07dc2bf42c5a1ba94feb3f543f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:57.845837 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e755baf9358248df98673ed087d7e8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:57.847295 8422 tablet_bootstrap.cc:492] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: Bootstrap starting.
I20260620 14:20:57.847501 8048 tablet_service.cc:1467] Processing CreateTablet for tablet e755baf9358248df98673ed087d7e8fd (DEFAULT_TABLE table=test-table [id=2826e07dc2bf42c5a1ba94feb3f543f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:57.847736 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e755baf9358248df98673ed087d7e8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:57.848315 8422 tablet_bootstrap.cc:654] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:57.849277 8445 ts_tablet_manager.cc:1905] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:57.850080 8423 tablet_bootstrap.cc:492] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: Bootstrap starting.
I20260620 14:20:57.850150 8424 tablet_bootstrap.cc:492] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: Bootstrap starting.
I20260620 14:20:57.850996 8423 tablet_bootstrap.cc:654] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:57.851132 8424 tablet_bootstrap.cc:654] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:57.851783 8445 ts_tablet_manager.cc:1918] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:57.851934 8446 log.cc:1196] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/1f81ff7cb373489b824ad9d99d5fe41c
I20260620 14:20:57.852336 8446 ts_tablet_manager.cc:1939] T 1f81ff7cb373489b824ad9d99d5fe41c P 6d72c244a5a44e6db348fad33b71a6e6: Deleting consensus metadata
I20260620 14:20:57.852424 8447 ts_tablet_manager.cc:1918] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:57.852596 8447 log.cc:1196] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/1f81ff7cb373489b824ad9d99d5fe41c
I20260620 14:20:57.852949 8447 ts_tablet_manager.cc:1939] T 1f81ff7cb373489b824ad9d99d5fe41c P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting consensus metadata
I20260620 14:20:57.853027 8423 tablet_bootstrap.cc:492] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: No bootstrap required, opened a new log
I20260620 14:20:57.853565 8423 ts_tablet_manager.cc:1397] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:20:57.853602 7895 catalog_manager.cc:4841] TS 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513): tablet 1f81ff7cb373489b824ad9d99d5fe41c (table test-table [id=7f82ffe37ca643648a4566c2984de97d]) successfully deleted
I20260620 14:20:57.854103 7896 catalog_manager.cc:4841] TS 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211): tablet 1f81ff7cb373489b824ad9d99d5fe41c (table test-table [id=7f82ffe37ca643648a4566c2984de97d]) successfully deleted
I20260620 14:20:57.854349 8423 raft_consensus.cc:348] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.854452 8423 raft_consensus.cc:374] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:57.854485 8423 raft_consensus.cc:729] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cd9547ed204da6aadcbba6ee230dd6, State: Initialized, Role: FOLLOWER
I20260620 14:20:57.854641 8424 tablet_bootstrap.cc:492] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: No bootstrap required, opened a new log
I20260620 14:20:57.854607 8423 consensus_queue.cc:260] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.854707 8424 ts_tablet_manager.cc:1397] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 14:20:57.854872 8423 ts_tablet_manager.cc:1428] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:20:57.855060 8445 log.cc:1196] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/1f81ff7cb373489b824ad9d99d5fe41c
I20260620 14:20:57.855206 8424 raft_consensus.cc:348] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.855321 8424 raft_consensus.cc:374] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:57.855365 8424 raft_consensus.cc:729] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d72c244a5a44e6db348fad33b71a6e6, State: Initialized, Role: FOLLOWER
I20260620 14:20:57.855404 8445 ts_tablet_manager.cc:1939] T 1f81ff7cb373489b824ad9d99d5fe41c P 74cd9547ed204da6aadcbba6ee230dd6: Deleting consensus metadata
I20260620 14:20:57.855502 8424 consensus_queue.cc:260] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.855751 8424 ts_tablet_manager.cc:1428] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:57.856686 7897 catalog_manager.cc:4841] TS 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779): tablet 1f81ff7cb373489b824ad9d99d5fe41c (table test-table [id=7f82ffe37ca643648a4566c2984de97d]) successfully deleted
I20260620 14:20:57.857925 8422 tablet_bootstrap.cc:492] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: No bootstrap required, opened a new log
I20260620 14:20:57.858039 8422 ts_tablet_manager.cc:1397] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260620 14:20:57.858503 8422 raft_consensus.cc:348] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.858606 8422 raft_consensus.cc:374] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:57.858639 8422 raft_consensus.cc:729] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9ffa7559154e4a8c98c1cc3db82795, State: Initialized, Role: FOLLOWER
I20260620 14:20:57.858760 8422 consensus_queue.cc:260] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.859025 8422 ts_tablet_manager.cc:1428] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:20:57.919119 8430 raft_consensus.cc:482] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:57.919312 8430 raft_consensus.cc:504] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.920495 8430 leader_election.cc:288] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211), 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513)
I20260620 14:20:57.925081 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.920838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33387 (local address 127.0.98.2:45211)
0620 14:20:57.920962 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.920968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.921044 (+ 76us) server_negotiation.cc:407] Connection header received
0620 14:20:57.921272 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.921278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.921335 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.921373 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:57.921858 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.922665 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.923863 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.924461 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.924570 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.924633 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.924895 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.924962 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:20:57.925107 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.920671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45211 (local address 127.0.98.1:33387)
0620 14:20:57.920960 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.920976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.921135 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.921515 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.921517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.921549 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:57.921722 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.921729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.922816 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.922819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.923655 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.923663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.924622 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.924628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.924753 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.924784 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.924875 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.924948 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":26,"spinlock_wait_cycles":24576,"thread_start_us":72,"threads_started":1}
I20260620 14:20:57.925671 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e755baf9358248df98673ed087d7e8fd" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" is_pre_election: true
I20260620 14:20:57.925927 8208 raft_consensus.cc:2454] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 0.
I20260620 14:20:57.926115 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.921138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41855 (local address 127.0.98.3:36513)
0620 14:20:57.921255 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.921261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.921276 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:20:57.921341 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.921344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.921392 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.921476 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:57.922841 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.923625 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.924766 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.925393 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.925497 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.925555 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.925924 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.926002 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:20:57.926378 8000 leader_election.cc:302] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9ffa7559154e4a8c98c1cc3db82795, 74cd9547ed204da6aadcbba6ee230dd6; no voters:
I20260620 14:20:57.926608 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.920758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36513 (local address 127.0.98.1:41855)
0620 14:20:57.921121 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.921133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.921208 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.921514 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.921517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.921543 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:57.921722 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.921729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.923802 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.923808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.924590 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.924598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.925673 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.925677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.925773 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.925801 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.926231 (+ 430us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.926315 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":27,"spinlock_wait_cycles":896,"thread_start_us":96,"threads_started":1}
I20260620 14:20:57.926672 8430 raft_consensus.cc:2781] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:57.926723 8430 raft_consensus.cc:482] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:57.926772 8430 raft_consensus.cc:3037] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:57.926966 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e755baf9358248df98673ed087d7e8fd" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d72c244a5a44e6db348fad33b71a6e6" is_pre_election: true
I20260620 14:20:57.927085 8347 raft_consensus.cc:2454] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 0.
I20260620 14:20:57.928033 8430 raft_consensus.cc:504] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.928386 8430 leader_election.cc:288] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211), 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513)
I20260620 14:20:57.928715 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e755baf9358248df98673ed087d7e8fd" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795"
I20260620 14:20:57.928733 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e755baf9358248df98673ed087d7e8fd" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d72c244a5a44e6db348fad33b71a6e6"
I20260620 14:20:57.928834 8208 raft_consensus.cc:3037] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:57.928840 8347 raft_consensus.cc:3037] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:57.929636 8208 raft_consensus.cc:2454] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 1.
I20260620 14:20:57.929890 8000 leader_election.cc:302] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9ffa7559154e4a8c98c1cc3db82795, 74cd9547ed204da6aadcbba6ee230dd6; no voters:
I20260620 14:20:57.929980 8347 raft_consensus.cc:2454] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 1.
I20260620 14:20:57.930087 8430 raft_consensus.cc:2781] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:57.930330 8430 raft_consensus.cc:686] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 LEADER]: Becoming Leader. State: Replica: 74cd9547ed204da6aadcbba6ee230dd6, State: Running, Role: LEADER
I20260620 14:20:57.930531 8430 consensus_queue.cc:237] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:57.932904 7911 catalog_manager.cc:5495] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "74cd9547ed204da6aadcbba6ee230dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } health_report { overall_health: UNKNOWN } } }
W20260620 14:20:57.945569 8395 tablet.cc:2369] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:20:57.987287 8255 tablet.cc:2369] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:20:57 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965256, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:57.991657 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.981651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.0.98.62:35719)
0620 14:20:57.981771 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:57.981775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:57.982035 (+ 260us) server_negotiation.cc:407] Connection header received
0620 14:20:57.982154 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:57.982158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:57.982209 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:57.982297 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:57.983059 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.983951 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.985090 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.985416 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.985521 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.988201 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:57.988205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:57.988207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:57.988230 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:57.989837 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.990354 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.990357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.990359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.990436 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:57.990713 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:57.990716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:57.990718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:57.990915 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:57.991001 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:57.991312 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:20:57.991486 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":50}
I20260620 14:20:57.991663 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:57.981574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35719 (local address 127.0.0.1:43640)
0620 14:20:57.981917 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:57.981932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:57.982054 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:57.982339 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:57.982342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:57.982789 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:57.982902 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.982909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.984059 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.984062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.984949 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:57.984958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:57.985539 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:57.985543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:57.985627 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:57.986258 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:57.986272 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:57.988062 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:57.989947 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.989956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.989962 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.990249 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.990536 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:57.990539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:57.990541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:57.990629 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:57.991013 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:57.991019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:57.991137 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 14:20:57.991253 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:20:57.991421 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260620 14:20:57.996024 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43640:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:20:57.996773 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:58.001423 8188 tablet_service.cc:1467] Processing CreateTablet for tablet c3ae9615caee4a3c90b51457dc9b4cc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc9f9353cab2448885caa1d8157f4e58]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:20:58.001423 8048 tablet_service.cc:1467] Processing CreateTablet for tablet c3ae9615caee4a3c90b51457dc9b4cc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc9f9353cab2448885caa1d8157f4e58]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:20:58.001827 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ae9615caee4a3c90b51457dc9b4cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:58.001834 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ae9615caee4a3c90b51457dc9b4cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:58.001716 8328 tablet_service.cc:1467] Processing CreateTablet for tablet c3ae9615caee4a3c90b51457dc9b4cc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc9f9353cab2448885caa1d8157f4e58]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:20:58.002110 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ae9615caee4a3c90b51457dc9b4cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:58.004755 8422 tablet_bootstrap.cc:492] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: Bootstrap starting.
I20260620 14:20:58.005065 8424 tablet_bootstrap.cc:492] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: Bootstrap starting.
I20260620 14:20:58.005694 8422 tablet_bootstrap.cc:654] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:58.006003 8424 tablet_bootstrap.cc:654] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:58.006232 8423 tablet_bootstrap.cc:492] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: Bootstrap starting.
I20260620 14:20:58.006799 8422 tablet_bootstrap.cc:492] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: No bootstrap required, opened a new log
I20260620 14:20:58.006904 8422 ts_tablet_manager.cc:1397] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:58.007266 8424 tablet_bootstrap.cc:492] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: No bootstrap required, opened a new log
I20260620 14:20:58.007313 8423 tablet_bootstrap.cc:654] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:58.007355 8424 ts_tablet_manager.cc:1397] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:20:58.007290 8422 raft_consensus.cc:348] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.007412 8422 raft_consensus.cc:374] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:58.007459 8422 raft_consensus.cc:729] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9ffa7559154e4a8c98c1cc3db82795, State: Initialized, Role: FOLLOWER
I20260620 14:20:58.007591 8422 consensus_queue.cc:260] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.007827 8424 raft_consensus.cc:348] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.007915 8422 ts_tablet_manager.cc:1428] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:58.007968 8424 raft_consensus.cc:374] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:58.007999 8424 raft_consensus.cc:729] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d72c244a5a44e6db348fad33b71a6e6, State: Initialized, Role: FOLLOWER
I20260620 14:20:58.008106 8424 consensus_queue.cc:260] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.007917 8440 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.008378 8424 ts_tablet_manager.cc:1428] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:20:58.008345 8431 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.008436 8440 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:58.008464 8431 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:58.009618 8423 tablet_bootstrap.cc:492] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: No bootstrap required, opened a new log
I20260620 14:20:58.009724 8423 ts_tablet_manager.cc:1397] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 14:20:58.010161 8423 raft_consensus.cc:348] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.010301 8423 raft_consensus.cc:374] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:58.010340 8423 raft_consensus.cc:729] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cd9547ed204da6aadcbba6ee230dd6, State: Initialized, Role: FOLLOWER
I20260620 14:20:58.010460 8423 consensus_queue.cc:260] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.010751 8423 ts_tablet_manager.cc:1428] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:20:58.010720 8430 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.010857 8430 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 14:20:58.024266 8115 tablet.cc:2369] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:58.218034 8431 raft_consensus.cc:482] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:58.218180 8431 raft_consensus.cc:504] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.219213 8431 leader_election.cc:288] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779), 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211)
I20260620 14:20:58.223210 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.219591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35525 (local address 127.0.98.2:45211)
0620 14:20:58.219731 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:58.219735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:58.219936 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:20:58.220101 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:58.220103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:58.220145 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:58.220224 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:58.220519 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.221069 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.222271 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.222650 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.222762 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.222833 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:58.223014 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:20:58.223084 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:20:58.223229 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.219470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45211 (local address 127.0.98.3:35525)
0620 14:20:58.219842 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:58.219854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:58.219980 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:58.220244 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:58.220247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:58.220263 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:58.220412 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.220418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.221177 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.221180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.222146 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.222154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.222765 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.222768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.222884 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.222910 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:20:58.222997 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:20:58.223075 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":56,"thread_start_us":74,"threads_started":1}
I20260620 14:20:58.223237 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.219578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49353 (local address 127.0.98.1:46779)
0620 14:20:58.219707 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:58.219712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:58.219978 (+ 266us) server_negotiation.cc:407] Connection header received
0620 14:20:58.220135 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:58.220138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:58.220185 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:58.220269 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:58.220569 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.221069 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.222243 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.222640 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.222760 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.222834 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:58.223045 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:20:58.223118 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:20:58.223253 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.219470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46779 (local address 127.0.98.3:49353)
0620 14:20:58.219882 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:58.219894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:58.220021 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:58.220279 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:58.220282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:58.220297 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:58.220461 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.220466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.221184 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.221187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.222104 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.222112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.222765 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.222768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.222925 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.222942 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:20:58.223029 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:20:58.223106 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":204,"thread_start_us":73,"threads_started":1}
I20260620 14:20:58.223645 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ae9615caee4a3c90b51457dc9b4cc4" candidate_uuid: "6d72c244a5a44e6db348fad33b71a6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" is_pre_election: true
I20260620 14:20:58.223843 8208 raft_consensus.cc:2454] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d72c244a5a44e6db348fad33b71a6e6 in term 0.
I20260620 14:20:58.224157 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ae9615caee4a3c90b51457dc9b4cc4" candidate_uuid: "6d72c244a5a44e6db348fad33b71a6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cd9547ed204da6aadcbba6ee230dd6" is_pre_election: true
I20260620 14:20:58.224293 8068 raft_consensus.cc:2454] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d72c244a5a44e6db348fad33b71a6e6 in term 0.
I20260620 14:20:58.224561 8280 leader_election.cc:302] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9ffa7559154e4a8c98c1cc3db82795, 6d72c244a5a44e6db348fad33b71a6e6; no voters:
I20260620 14:20:58.224821 8431 raft_consensus.cc:2781] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:58.224879 8431 raft_consensus.cc:482] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:58.224925 8431 raft_consensus.cc:3037] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:58.225979 8431 raft_consensus.cc:504] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.226290 8431 leader_election.cc:288] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [CANDIDATE]: Term 1 election: Requested vote from peers 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779), 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211)
I20260620 14:20:58.226577 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ae9615caee4a3c90b51457dc9b4cc4" candidate_uuid: "6d72c244a5a44e6db348fad33b71a6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cd9547ed204da6aadcbba6ee230dd6"
I20260620 14:20:58.226622 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ae9615caee4a3c90b51457dc9b4cc4" candidate_uuid: "6d72c244a5a44e6db348fad33b71a6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795"
I20260620 14:20:58.226681 8068 raft_consensus.cc:3037] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:58.226765 8208 raft_consensus.cc:3037] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:58.227562 8068 raft_consensus.cc:2454] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d72c244a5a44e6db348fad33b71a6e6 in term 1.
I20260620 14:20:58.227715 8208 raft_consensus.cc:2454] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d72c244a5a44e6db348fad33b71a6e6 in term 1.
I20260620 14:20:58.227876 8281 leader_election.cc:302] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d72c244a5a44e6db348fad33b71a6e6, 74cd9547ed204da6aadcbba6ee230dd6; no voters:
I20260620 14:20:58.228091 8431 raft_consensus.cc:2781] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:58.228231 8431 raft_consensus.cc:686] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 LEADER]: Becoming Leader. State: Replica: 6d72c244a5a44e6db348fad33b71a6e6, State: Running, Role: LEADER
I20260620 14:20:58.228502 8431 consensus_queue.cc:237] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.229560 8466 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: TxnStatusTablet state changed. Reason: New leader 6d72c244a5a44e6db348fad33b71a6e6. Latest consensus state: current_term: 1 leader_uuid: "6d72c244a5a44e6db348fad33b71a6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.229669 8466 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:20:58.230157 8468 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:20:58.230314 8468 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:20:58.230803 7911 catalog_manager.cc:5495] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6d72c244a5a44e6db348fad33b71a6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } health_report { overall_health: HEALTHY } } }
I20260620 14:20:58.245978 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.242119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36513 (local address 127.0.0.1:43340)
0620 14:20:58.242289 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:58.242305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:58.242426 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:58.242746 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:58.242750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:58.242760 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:58.242864 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.242870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.243629 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.243632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.244331 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.244340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.244772 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.244775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.244886 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.244926 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:58.245719 (+ 793us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:58.245731 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:20:58.245789 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:20:58.245864 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:20:58.246008 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.242221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.0.98.3:36513)
0620 14:20:58.242333 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:58.242340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:58.242357 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:20:58.242563 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:58.242567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:58.242628 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:58.242725 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:58.242970 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.243518 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.244454 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.244668 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.244772 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.245119 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:58.245620 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:58.245739 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:58.245818 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:20:58.245876 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:20:58.248803 8068 raft_consensus.cc:1261] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Refusing update from remote peer 6d72c244a5a44e6db348fad33b71a6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:58.249081 8208 raft_consensus.cc:1261] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Refusing update from remote peer 6d72c244a5a44e6db348fad33b71a6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:58.249306 8466 consensus_queue.cc:1035] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:58.249571 8431 consensus_queue.cc:1035] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:58.251053 8440 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: TxnStatusTablet state changed. Reason: New leader 6d72c244a5a44e6db348fad33b71a6e6. Latest consensus state: current_term: 1 leader_uuid: "6d72c244a5a44e6db348fad33b71a6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.251174 8440 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:58.252836 8430 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: TxnStatusTablet state changed. Reason: New leader 6d72c244a5a44e6db348fad33b71a6e6. Latest consensus state: current_term: 1 leader_uuid: "6d72c244a5a44e6db348fad33b71a6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.252957 8430 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:58.254437 8466 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d72c244a5a44e6db348fad33b71a6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.254547 8431 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d72c244a5a44e6db348fad33b71a6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.254642 8431 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:20:58.254551 8466 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 6d72c244a5a44e6db348fad33b71a6e6: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:20:58.255699 8430 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d72c244a5a44e6db348fad33b71a6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.255839 8430 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 74cd9547ed204da6aadcbba6ee230dd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:58.256629 8440 tablet_replica.cc:439] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d72c244a5a44e6db348fad33b71a6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } } }
I20260620 14:20:58.256729 8440 tablet_replica.cc:442] T c3ae9615caee4a3c90b51457dc9b4cc4 P 0c9ffa7559154e4a8c98c1cc3db82795: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:20:58.263628 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.259969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.0.98.1:46779)
0620 14:20:58.260075 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:58.260079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:58.260150 (+ 71us) server_negotiation.cc:407] Connection header received
0620 14:20:58.260350 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:58.260353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:58.260411 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:58.260538 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:58.260871 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.261376 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.262217 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.262455 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.262564 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.262825 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:58.263247 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:58.263375 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:58.263438 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:20:58.263490 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:20:58.275154 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.270843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36513 (local address 127.0.98.1:44859)
0620 14:20:58.271162 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:58.271179 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:58.271333 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:58.271706 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:58.271709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:58.271717 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:58.271876 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.271885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.272646 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.272653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.273879 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.273889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.274373 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.274377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.274478 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.274521 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:58.274831 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:58.274838 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:20:58.274894 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:20:58.274972 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260620 14:20:58.275504 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.270995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44859 (local address 127.0.98.3:36513)
0620 14:20:58.271128 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:58.271138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:58.271261 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:20:58.271456 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:58.271459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:58.271513 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:58.271661 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:58.272003 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.272524 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.274022 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.274258 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.274374 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.274658 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:58.274737 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:58.274818 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:58.275269 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 14:20:58.275349 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:20:58.282898 8347 raft_consensus.cc:1261] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Refusing update from remote peer 74cd9547ed204da6aadcbba6ee230dd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:58.283021 8207 raft_consensus.cc:1261] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Refusing update from remote peer 74cd9547ed204da6aadcbba6ee230dd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:58.283550 8430 consensus_queue.cc:1035] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:58.283834 8453 consensus_queue.cc:1035] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:58.290946 8435 mvcc.cc:205] Tried to move back new op lower bound from 7298929697921871872 to 7298929696485867520. Current Snapshot: MvccSnapshot[applied={T|T < 7298929697921871872}]
I20260620 14:20:58.313216 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.309086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45615 (local address 127.0.98.1:46779)
0620 14:20:58.309231 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:58.309237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:58.309482 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:20:58.309648 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:58.309651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:58.309695 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:58.309768 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:58.310067 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.310656 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.312046 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.312392 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.312484 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.312809 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:58.312876 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:58.312936 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:58.313048 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:20:58.313106 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:20:58.313287 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.309027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46779 (local address 127.0.98.3:45615)
0620 14:20:58.309386 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:58.309402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:58.309509 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:58.309799 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:58.309802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:58.309809 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:20:58.309929 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.309935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.310768 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.310771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.311803 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:58.311815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.312525 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.312529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:58.312626 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.312678 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:20:58.312963 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:20:58.312969 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:20:58.313041 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:20:58.313126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":145,"threads_started":1}
I20260620 14:20:58.343609 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:58.343828 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:58.346166 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: Sending DeleteTablet for 3 replicas of tablet e755baf9358248df98673ed087d7e8fd
I20260620 14:20:58.346906 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet e755baf9358248df98673ed087d7e8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50294
I20260620 14:20:58.346937 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet e755baf9358248df98673ed087d7e8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43330
I20260620 14:20:58.347143 8445 tablet_replica.cc:330] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: stopping tablet replica
I20260620 14:20:58.347260 8445 raft_consensus.cc:2229] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:58.347265 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet e755baf9358248df98673ed087d7e8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40570
I20260620 14:20:58.347442 8447 tablet_replica.cc:330] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: stopping tablet replica
I20260620 14:20:58.347539 8445 raft_consensus.cc:2258] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:58.347550 8447 raft_consensus.cc:2229] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:58.347657 8447 raft_consensus.cc:2258] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:58.347725 8446 tablet_replica.cc:330] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: stopping tablet replica
I20260620 14:20:58.348030 8446 raft_consensus.cc:2229] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:58.348160 8446 raft_consensus.cc:2258] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:58.349244 8447 ts_tablet_manager.cc:1905] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:58.349268 8446 ts_tablet_manager.cc:1905] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:58.351567 8445 ts_tablet_manager.cc:1905] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:58.352386 8446 ts_tablet_manager.cc:1918] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:58.352468 8446 log.cc:1196] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/e755baf9358248df98673ed087d7e8fd
I20260620 14:20:58.352416 8447 ts_tablet_manager.cc:1918] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:58.352669 8447 log.cc:1196] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/e755baf9358248df98673ed087d7e8fd
I20260620 14:20:58.352722 8446 ts_tablet_manager.cc:1939] T e755baf9358248df98673ed087d7e8fd P 6d72c244a5a44e6db348fad33b71a6e6: Deleting consensus metadata
I20260620 14:20:58.353014 8447 ts_tablet_manager.cc:1939] T e755baf9358248df98673ed087d7e8fd P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting consensus metadata
I20260620 14:20:58.353538 7895 catalog_manager.cc:4841] TS 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513): tablet e755baf9358248df98673ed087d7e8fd (table test-table [id=2826e07dc2bf42c5a1ba94feb3f543f5]) successfully deleted
I20260620 14:20:58.354221 7896 catalog_manager.cc:4841] TS 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211): tablet e755baf9358248df98673ed087d7e8fd (table test-table [id=2826e07dc2bf42c5a1ba94feb3f543f5]) successfully deleted
I20260620 14:20:58.354342 8445 ts_tablet_manager.cc:1918] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:58.354422 8445 log.cc:1196] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/e755baf9358248df98673ed087d7e8fd
I20260620 14:20:58.354686 8445 ts_tablet_manager.cc:1939] T e755baf9358248df98673ed087d7e8fd P 74cd9547ed204da6aadcbba6ee230dd6: Deleting consensus metadata
I20260620 14:20:58.355415 7897 catalog_manager.cc:4841] TS 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779): tablet e755baf9358248df98673ed087d7e8fd (table test-table [id=2826e07dc2bf42c5a1ba94feb3f543f5]) successfully deleted
Jun 20 14:20:58 dist-test-slave-87nc krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:20:58 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:20:58 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:20:58.385840 7960 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:20:58.377718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.0.98.62:35719)
0620 14:20:58.377855 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:58.377860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:58.378146 (+ 286us) server_negotiation.cc:407] Connection header received
0620 14:20:58.378284 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:58.378287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:58.378345 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:58.378437 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:58.379379 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.380247 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.381480 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.381835 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.381942 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.385701 (+ 3759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43644: BlockingRecv error: recv got EOF from 127.0.0.1:43644 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:20:58 dist-test-slave-87nc krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:20:58.398434 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.387954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.0.98.62:35719)
0620 14:20:58.388057 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:58.388061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:58.388494 (+ 433us) server_negotiation.cc:407] Connection header received
0620 14:20:58.388612 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:58.388616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:58.388664 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:58.388747 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:58.389481 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.390286 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.391460 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.391866 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.391967 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.394795 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:58.394800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:58.394803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:58.394841 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:58.396623 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:58.397088 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:58.397092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:58.397094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:58.397185 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:58.397500 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:58.397503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:58.397505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:58.397748 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:58.397888 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:58.398130 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:20:58.398255 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:20:58.401402 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:58.401854 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:58.407439 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9c45f1894947469624dedb8bd07b17 (DEFAULT_TABLE table=test-table [id=fa472fc9000f446c81033adff962c7c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:58.407753 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9c45f1894947469624dedb8bd07b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:58.407922 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9c45f1894947469624dedb8bd07b17 (DEFAULT_TABLE table=test-table [id=fa472fc9000f446c81033adff962c7c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:58.408193 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9c45f1894947469624dedb8bd07b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:58.410336 8423 tablet_bootstrap.cc:492] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: Bootstrap starting.
I20260620 14:20:58.410586 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9c45f1894947469624dedb8bd07b17 (DEFAULT_TABLE table=test-table [id=fa472fc9000f446c81033adff962c7c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:58.410825 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9c45f1894947469624dedb8bd07b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:58.411269 8423 tablet_bootstrap.cc:654] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:58.413107 8422 tablet_bootstrap.cc:492] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: Bootstrap starting.
I20260620 14:20:58.413940 8422 tablet_bootstrap.cc:654] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:58.414427 8424 tablet_bootstrap.cc:492] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: Bootstrap starting.
I20260620 14:20:58.415180 8424 tablet_bootstrap.cc:654] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:58.417428 8424 tablet_bootstrap.cc:492] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: No bootstrap required, opened a new log
I20260620 14:20:58.417513 8424 ts_tablet_manager.cc:1397] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:20:58.417909 8424 raft_consensus.cc:348] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.418016 8424 raft_consensus.cc:374] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:58.418042 8424 raft_consensus.cc:729] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d72c244a5a44e6db348fad33b71a6e6, State: Initialized, Role: FOLLOWER
I20260620 14:20:58.418170 8424 consensus_queue.cc:260] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.418416 8424 ts_tablet_manager.cc:1428] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:58.419420 8422 tablet_bootstrap.cc:492] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: No bootstrap required, opened a new log
I20260620 14:20:58.419504 8422 ts_tablet_manager.cc:1397] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 14:20:58.419907 8422 raft_consensus.cc:348] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.420012 8422 raft_consensus.cc:374] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:58.420040 8422 raft_consensus.cc:729] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9ffa7559154e4a8c98c1cc3db82795, State: Initialized, Role: FOLLOWER
I20260620 14:20:58.420164 8422 consensus_queue.cc:260] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.420413 8422 ts_tablet_manager.cc:1428] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:58.422026 8423 tablet_bootstrap.cc:492] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: No bootstrap required, opened a new log
I20260620 14:20:58.422122 8423 ts_tablet_manager.cc:1397] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20260620 14:20:58.422577 8423 raft_consensus.cc:348] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.422700 8423 raft_consensus.cc:374] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:58.422731 8423 raft_consensus.cc:729] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cd9547ed204da6aadcbba6ee230dd6, State: Initialized, Role: FOLLOWER
I20260620 14:20:58.422855 8423 consensus_queue.cc:260] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.423105 8423 ts_tablet_manager.cc:1428] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:20:58.486037 8466 raft_consensus.cc:482] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:58.486179 8466 raft_consensus.cc:504] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.486428 8484 raft_consensus.cc:482] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:58.486521 8484 raft_consensus.cc:504] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.486668 8466 leader_election.cc:288] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779), 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211)
I20260620 14:20:58.486867 8484 leader_election.cc:288] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211), 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513)
I20260620 14:20:58.487223 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9c45f1894947469624dedb8bd07b17" candidate_uuid: "6d72c244a5a44e6db348fad33b71a6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cd9547ed204da6aadcbba6ee230dd6" is_pre_election: true
I20260620 14:20:58.487260 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9c45f1894947469624dedb8bd07b17" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d72c244a5a44e6db348fad33b71a6e6" is_pre_election: true
I20260620 14:20:58.487354 8347 raft_consensus.cc:2454] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 0.
I20260620 14:20:58.487373 8068 raft_consensus.cc:2454] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d72c244a5a44e6db348fad33b71a6e6 in term 0.
I20260620 14:20:58.487675 8281 leader_election.cc:302] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d72c244a5a44e6db348fad33b71a6e6, 74cd9547ed204da6aadcbba6ee230dd6; no voters:
I20260620 14:20:58.487804 7999 leader_election.cc:302] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d72c244a5a44e6db348fad33b71a6e6, 74cd9547ed204da6aadcbba6ee230dd6; no voters:
I20260620 14:20:58.487874 8466 raft_consensus.cc:2781] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:58.487918 8466 raft_consensus.cc:482] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:58.487963 8466 raft_consensus.cc:3037] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:58.488006 8484 raft_consensus.cc:2781] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:58.488047 8484 raft_consensus.cc:482] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:58.488083 8484 raft_consensus.cc:3037] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:58.488256 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9c45f1894947469624dedb8bd07b17" candidate_uuid: "6d72c244a5a44e6db348fad33b71a6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" is_pre_election: true
I20260620 14:20:58.488409 8208 raft_consensus.cc:2454] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d72c244a5a44e6db348fad33b71a6e6 in term 0.
I20260620 14:20:58.489056 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9c45f1894947469624dedb8bd07b17" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" is_pre_election: true
I20260620 14:20:58.489185 8208 raft_consensus.cc:2454] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 0.
I20260620 14:20:58.489219 8484 raft_consensus.cc:504] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.489218 8466 raft_consensus.cc:504] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.489529 8484 leader_election.cc:288] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211), 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513)
I20260620 14:20:58.489537 8466 leader_election.cc:288] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [CANDIDATE]: Term 1 election: Requested vote from peers 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779), 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211)
I20260620 14:20:58.489869 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9c45f1894947469624dedb8bd07b17" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795"
I20260620 14:20:58.489974 8208 raft_consensus.cc:3037] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:58.490273 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9c45f1894947469624dedb8bd07b17" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d72c244a5a44e6db348fad33b71a6e6"
I20260620 14:20:58.490285 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9c45f1894947469624dedb8bd07b17" candidate_uuid: "6d72c244a5a44e6db348fad33b71a6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795"
I20260620 14:20:58.490417 8347 raft_consensus.cc:2379] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 74cd9547ed204da6aadcbba6ee230dd6 in current term 1: Already voted for candidate 6d72c244a5a44e6db348fad33b71a6e6 in this term.
I20260620 14:20:58.490783 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9c45f1894947469624dedb8bd07b17" candidate_uuid: "6d72c244a5a44e6db348fad33b71a6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cd9547ed204da6aadcbba6ee230dd6"
I20260620 14:20:58.490934 8068 raft_consensus.cc:2379] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d72c244a5a44e6db348fad33b71a6e6 in current term 1: Already voted for candidate 74cd9547ed204da6aadcbba6ee230dd6 in this term.
I20260620 14:20:58.491214 8208 raft_consensus.cc:2454] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 1.
I20260620 14:20:58.491216 8281 leader_election.cc:302] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d72c244a5a44e6db348fad33b71a6e6; no voters: 0c9ffa7559154e4a8c98c1cc3db82795, 74cd9547ed204da6aadcbba6ee230dd6
I20260620 14:20:58.491446 8466 raft_consensus.cc:2726] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 14:20:58.491532 8000 leader_election.cc:302] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c9ffa7559154e4a8c98c1cc3db82795, 74cd9547ed204da6aadcbba6ee230dd6; no voters: 6d72c244a5a44e6db348fad33b71a6e6
I20260620 14:20:58.491732 8484 raft_consensus.cc:2781] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:58.491847 8484 raft_consensus.cc:686] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 LEADER]: Becoming Leader. State: Replica: 74cd9547ed204da6aadcbba6ee230dd6, State: Running, Role: LEADER
I20260620 14:20:58.491964 8484 consensus_queue.cc:237] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.493505 7910 catalog_manager.cc:5495] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "74cd9547ed204da6aadcbba6ee230dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } health_report { overall_health: UNKNOWN } } }
I20260620 14:20:58.506647 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.502738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.0.98.1:46779)
0620 14:20:58.502845 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:58.502849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:58.502948 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:20:58.503089 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:58.503092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:58.503140 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:58.503218 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:58.503437 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.504292 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.505496 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:58.505835 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:58.505935 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:58.506203 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:58.506292 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:58.506362 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:58.506461 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 14:20:58.506513 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:20:58.508704 8347 raft_consensus.cc:1261] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Refusing update from remote peer 74cd9547ed204da6aadcbba6ee230dd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:58.508698 8208 raft_consensus.cc:1261] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Refusing update from remote peer 74cd9547ed204da6aadcbba6ee230dd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:58.509280 8484 consensus_queue.cc:1035] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:58.509513 8430 consensus_queue.cc:1035] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:58.520565 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:58.520700 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:58.522485 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: Sending DeleteTablet for 3 replicas of tablet 9c9c45f1894947469624dedb8bd07b17
I20260620 14:20:58.522970 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9c45f1894947469624dedb8bd07b17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50294
I20260620 14:20:58.523036 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9c45f1894947469624dedb8bd07b17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43330
I20260620 14:20:58.523046 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9c45f1894947469624dedb8bd07b17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40570
I20260620 14:20:58.523164 8445 tablet_replica.cc:330] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: stopping tablet replica
I20260620 14:20:58.523176 8446 tablet_replica.cc:330] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: stopping tablet replica
I20260620 14:20:58.523214 8447 tablet_replica.cc:330] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: stopping tablet replica
I20260620 14:20:58.523269 8445 raft_consensus.cc:2229] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:58.523269 8446 raft_consensus.cc:2229] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:58.523317 8447 raft_consensus.cc:2229] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:58.523396 8446 raft_consensus.cc:2258] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:58.523425 8447 raft_consensus.cc:2258] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:58.523444 8445 raft_consensus.cc:2258] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:58.524206 8445 ts_tablet_manager.cc:1905] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:58.524305 8446 ts_tablet_manager.cc:1905] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:58.524780 8447 ts_tablet_manager.cc:1905] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:58.526538 8445 ts_tablet_manager.cc:1918] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:58.526538 8446 ts_tablet_manager.cc:1918] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:58.526607 8445 log.cc:1196] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/9c9c45f1894947469624dedb8bd07b17
I20260620 14:20:58.526607 8446 log.cc:1196] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/9c9c45f1894947469624dedb8bd07b17
I20260620 14:20:58.526865 8445 ts_tablet_manager.cc:1939] T 9c9c45f1894947469624dedb8bd07b17 P 74cd9547ed204da6aadcbba6ee230dd6: Deleting consensus metadata
I20260620 14:20:58.526865 8446 ts_tablet_manager.cc:1939] T 9c9c45f1894947469624dedb8bd07b17 P 6d72c244a5a44e6db348fad33b71a6e6: Deleting consensus metadata
I20260620 14:20:58.526975 8447 ts_tablet_manager.cc:1918] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:58.527038 8447 log.cc:1196] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/9c9c45f1894947469624dedb8bd07b17
I20260620 14:20:58.527295 8447 ts_tablet_manager.cc:1939] T 9c9c45f1894947469624dedb8bd07b17 P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting consensus metadata
I20260620 14:20:58.527534 7897 catalog_manager.cc:4841] TS 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779): tablet 9c9c45f1894947469624dedb8bd07b17 (table test-table [id=fa472fc9000f446c81033adff962c7c3]) successfully deleted
I20260620 14:20:58.527827 7895 catalog_manager.cc:4841] TS 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513): tablet 9c9c45f1894947469624dedb8bd07b17 (table test-table [id=fa472fc9000f446c81033adff962c7c3]) successfully deleted
I20260620 14:20:58.528024 7896 catalog_manager.cc:4841] TS 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211): tablet 9c9c45f1894947469624dedb8bd07b17 (table test-table [id=fa472fc9000f446c81033adff962c7c3]) successfully deleted
I20260620 14:20:58.528079 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:58.528524 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:58.532910 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 4722146cd73343abba25f547bdb64a19 (DEFAULT_TABLE table=test-table [id=ae5f23848cd64a79865430e57ab09028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:58.532985 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 4722146cd73343abba25f547bdb64a19 (DEFAULT_TABLE table=test-table [id=ae5f23848cd64a79865430e57ab09028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:58.533051 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 4722146cd73343abba25f547bdb64a19 (DEFAULT_TABLE table=test-table [id=ae5f23848cd64a79865430e57ab09028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:58.533196 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4722146cd73343abba25f547bdb64a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:58.533232 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4722146cd73343abba25f547bdb64a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:58.533324 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4722146cd73343abba25f547bdb64a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:58.535526 8424 tablet_bootstrap.cc:492] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: Bootstrap starting.
I20260620 14:20:58.535717 8423 tablet_bootstrap.cc:492] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6: Bootstrap starting.
I20260620 14:20:58.535904 8422 tablet_bootstrap.cc:492] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: Bootstrap starting.
I20260620 14:20:58.536444 8424 tablet_bootstrap.cc:654] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:58.536669 8422 tablet_bootstrap.cc:654] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:58.537391 8423 tablet_bootstrap.cc:654] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:58.537410 8424 tablet_bootstrap.cc:492] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: No bootstrap required, opened a new log
I20260620 14:20:58.537508 8424 ts_tablet_manager.cc:1397] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:58.537879 8424 raft_consensus.cc:348] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.538007 8424 raft_consensus.cc:374] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:58.538045 8424 raft_consensus.cc:729] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d72c244a5a44e6db348fad33b71a6e6, State: Initialized, Role: FOLLOWER
I20260620 14:20:58.538188 8424 consensus_queue.cc:260] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.538515 8424 ts_tablet_manager.cc:1428] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:58.538539 8422 tablet_bootstrap.cc:492] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: No bootstrap required, opened a new log
I20260620 14:20:58.538563 8423 tablet_bootstrap.cc:492] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6: No bootstrap required, opened a new log
I20260620 14:20:58.538615 8422 ts_tablet_manager.cc:1397] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:20:58.538630 8423 ts_tablet_manager.cc:1397] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 14:20:58.539003 8422 raft_consensus.cc:348] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.539043 8423 raft_consensus.cc:348] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.539134 8422 raft_consensus.cc:374] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:58.539147 8423 raft_consensus.cc:374] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:58.539170 8422 raft_consensus.cc:729] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9ffa7559154e4a8c98c1cc3db82795, State: Initialized, Role: FOLLOWER
I20260620 14:20:58.539191 8423 raft_consensus.cc:729] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cd9547ed204da6aadcbba6ee230dd6, State: Initialized, Role: FOLLOWER
I20260620 14:20:58.539283 8422 consensus_queue.cc:260] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.539302 8423 consensus_queue.cc:260] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.539569 8423 ts_tablet_manager.cc:1428] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:20:58.539566 8422 ts_tablet_manager.cc:1428] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:20:58.557173 8479 raft_consensus.cc:482] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:58.557317 8479 raft_consensus.cc:504] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.557765 8479 leader_election.cc:288] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211), 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513)
I20260620 14:20:58.558182 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4722146cd73343abba25f547bdb64a19" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d72c244a5a44e6db348fad33b71a6e6" is_pre_election: true
I20260620 14:20:58.558177 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4722146cd73343abba25f547bdb64a19" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" is_pre_election: true
I20260620 14:20:58.558372 8347 raft_consensus.cc:2454] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 0.
I20260620 14:20:58.558364 8208 raft_consensus.cc:2454] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 0.
I20260620 14:20:58.558647 8000 leader_election.cc:302] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9ffa7559154e4a8c98c1cc3db82795, 74cd9547ed204da6aadcbba6ee230dd6; no voters:
I20260620 14:20:58.558835 8479 raft_consensus.cc:2781] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:58.558903 8479 raft_consensus.cc:482] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:58.558929 8479 raft_consensus.cc:3037] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:58.560003 8479 raft_consensus.cc:504] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.560329 8479 leader_election.cc:288] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211), 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513)
I20260620 14:20:58.560657 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4722146cd73343abba25f547bdb64a19" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ffa7559154e4a8c98c1cc3db82795"
I20260620 14:20:58.560658 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4722146cd73343abba25f547bdb64a19" candidate_uuid: "74cd9547ed204da6aadcbba6ee230dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d72c244a5a44e6db348fad33b71a6e6"
I20260620 14:20:58.560779 8208 raft_consensus.cc:3037] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:58.560804 8347 raft_consensus.cc:3037] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:58.561846 8208 raft_consensus.cc:2454] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 1.
I20260620 14:20:58.561856 8347 raft_consensus.cc:2454] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74cd9547ed204da6aadcbba6ee230dd6 in term 1.
I20260620 14:20:58.562106 8000 leader_election.cc:302] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9ffa7559154e4a8c98c1cc3db82795, 74cd9547ed204da6aadcbba6ee230dd6; no voters:
I20260620 14:20:58.562292 8479 raft_consensus.cc:2781] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:58.562462 8479 raft_consensus.cc:686] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 LEADER]: Becoming Leader. State: Replica: 74cd9547ed204da6aadcbba6ee230dd6, State: Running, Role: LEADER
I20260620 14:20:58.562640 8479 consensus_queue.cc:237] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } }
I20260620 14:20:58.564185 7911 catalog_manager.cc:5495] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "74cd9547ed204da6aadcbba6ee230dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74cd9547ed204da6aadcbba6ee230dd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 } health_report { overall_health: UNKNOWN } } }
I20260620 14:20:58.586265 8208 raft_consensus.cc:1261] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Refusing update from remote peer 74cd9547ed204da6aadcbba6ee230dd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:58.586269 8347 raft_consensus.cc:1261] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Refusing update from remote peer 74cd9547ed204da6aadcbba6ee230dd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:58.586684 8453 consensus_queue.cc:1035] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9ffa7559154e4a8c98c1cc3db82795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:58.586869 8479 consensus_queue.cc:1035] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d72c244a5a44e6db348fad33b71a6e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:58.616281 7911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:58.616422 7911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:58.618157 7911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1820151c77a409c996d34e4af7b4232: Sending DeleteTablet for 3 replicas of tablet 4722146cd73343abba25f547bdb64a19
I20260620 14:20:58.618723 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 4722146cd73343abba25f547bdb64a19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50294
I20260620 14:20:58.618860 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 4722146cd73343abba25f547bdb64a19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40570
I20260620 14:20:58.618916 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet 4722146cd73343abba25f547bdb64a19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43330
I20260620 14:20:58.619057 8446 tablet_replica.cc:330] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: stopping tablet replica
I20260620 14:20:58.619060 8445 tablet_replica.cc:330] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6: stopping tablet replica
I20260620 14:20:58.619174 8445 raft_consensus.cc:2229] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:58.619172 8446 raft_consensus.cc:2229] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:58.619292 8446 raft_consensus.cc:2258] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:58.619372 8445 raft_consensus.cc:2258] T 4722146cd73343abba25f547bdb64a19 P 74cd9547ed204da6aadcbba6ee230dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:58.619792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 7983
I20260620 14:20:58.619952 8447 tablet_replica.cc:330] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: stopping tablet replica
I20260620 14:20:58.620090 8447 raft_consensus.cc:2229] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:58.620188 8447 raft_consensus.cc:2258] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:58.620761 8446 ts_tablet_manager.cc:1905] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:58.622579 8447 ts_tablet_manager.cc:1905] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:58.622797 8446 ts_tablet_manager.cc:1918] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:58.622889 8446 log.cc:1196] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/4722146cd73343abba25f547bdb64a19
I20260620 14:20:58.623118 8446 ts_tablet_manager.cc:1939] T 4722146cd73343abba25f547bdb64a19 P 6d72c244a5a44e6db348fad33b71a6e6: Deleting consensus metadata
I20260620 14:20:58.623805 7895 catalog_manager.cc:4841] TS 6d72c244a5a44e6db348fad33b71a6e6 (127.0.98.3:36513): tablet 4722146cd73343abba25f547bdb64a19 (table test-table [id=ae5f23848cd64a79865430e57ab09028]) successfully deleted
I20260620 14:20:58.625774 8447 ts_tablet_manager.cc:1918] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:20:58.626603 8447 log.cc:1196] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/4722146cd73343abba25f547bdb64a19
I20260620 14:20:58.626969 8447 ts_tablet_manager.cc:1939] T 4722146cd73343abba25f547bdb64a19 P 0c9ffa7559154e4a8c98c1cc3db82795: Deleting consensus metadata
I20260620 14:20:58.628228 7896 catalog_manager.cc:4841] TS 0c9ffa7559154e4a8c98c1cc3db82795 (127.0.98.2:45211): tablet 4722146cd73343abba25f547bdb64a19 (table test-table [id=ae5f23848cd64a79865430e57ab09028]) successfully deleted
W20260620 14:20:58.633180 7897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46779 (error 108)
I20260620 14:20:58.633397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 8123
I20260620 14:20:58.634796 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:58.634370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46779 (local address 127.0.0.1:50322)
0620 14:20:58.634618 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:58.634696 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":55,"threads_started":1}
W20260620 14:20:58.635905 7897 catalog_manager.cc:4584] TS 74cd9547ed204da6aadcbba6ee230dd6 (127.0.98.1:46779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4722146cd73343abba25f547bdb64a19: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46779: connect: Connection refused (error 111)
I20260620 14:20:58.641755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 8263
I20260620 14:20:58.650606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 7879
2026-06-20T14:20:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2997 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:58 dist-test-slave-87nc krb5kdc[8515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:58 dist-test-slave-87nc krb5kdc[8515](info): set up 2 sockets
Jun 20 14:20:58 dist-test-slave-87nc krb5kdc[8515](info): commencing operation
krb5kdc: starting...
W20260620 14:20:59.688673 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:59 dist-test-slave-87nc krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:59.806813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:59.912703 8528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:59.912968 8528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:59.913008 8528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:59.917356 8528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:59.917433 8528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:59.917450 8528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:59.917482 8528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:59.917497 8528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:59.922794 8528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:20:59.923843 8528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:59.924896 8528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:59.933425 8534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:59.933482 8536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:59.933630 8528 server_base.cc:1034] running on GCE node
W20260620 14:20:59.933444 8533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:59.934481 8528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:59.935631 8528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:59.936823 8528 hybrid_clock.cc:648] HybridClock initialized: now 1781965259936811 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:20:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1292 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:21:00 dist-test-slave-87nc krb5kdc[8541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:21:00 dist-test-slave-87nc krb5kdc[8541](info): set up 2 sockets
Jun 20 14:21:00 dist-test-slave-87nc krb5kdc[8541](info): commencing operation
krb5kdc: starting...
W20260620 14:21:00.984563 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:21:01 dist-test-slave-87nc krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:21:01.108934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43089
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:01.216773 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:01.217101 8554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:01.217190 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:01.221292 8554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:01.221395 8554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:01.221434 8554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:01.221472 8554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:01.221503 8554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:01.226864 8554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:01.228016 8554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:01.229072 8554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:01.237774 8560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:01.237732 8559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:01.237721 8562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:01.237862 8554 server_base.cc:1034] running on GCE node
I20260620 14:21:01.238735 8554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:01.240023 8554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:01.241205 8554 hybrid_clock.cc:648] HybridClock initialized: now 1781965261241192 us; error 28 us; skew 500 ppm
Jun 20 14:21:01 dist-test-slave-87nc krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:01.247622 8554 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:01.248570 8554 webserver.cc:460] Webserver started at http://127.0.98.62:38489/ using document root <none> and password file <none>
I20260620 14:21:01.249141 8554 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:01.249229 8554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:01.249460 8554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:01.251539 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "ec52afbbfa0d478fbe0fe45ea63d919b"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "81354be05dd438170fbd132fb70222cb"
server_key_iv: "123d69cf359ebfdbc1713567e06a7b1b"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.252116 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec52afbbfa0d478fbe0fe45ea63d919b"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "81354be05dd438170fbd132fb70222cb"
server_key_iv: "123d69cf359ebfdbc1713567e06a7b1b"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.255712 8554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:21:01.258217 8569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:01.259442 8554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:01.259591 8554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "ec52afbbfa0d478fbe0fe45ea63d919b"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "81354be05dd438170fbd132fb70222cb"
server_key_iv: "123d69cf359ebfdbc1713567e06a7b1b"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.259727 8554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:01.273630 8554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:01.276719 8554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:01.276937 8554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:01.284082 8554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43089
I20260620 14:21:01.284087 8621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43089 every 8 connection(s)
I20260620 14:21:01.285245 8554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:01.288026 8622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:01.293594 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: Bootstrap starting.
I20260620 14:21:01.295368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 8554
I20260620 14:21:01.295533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:21:01.295914 392 external_mini_cluster.cc:1428] Setting key ab1f61ca77fe123d259739059d2808e1
I20260620 14:21:01.296196 8622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:01.297144 8622 log.cc:824] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:01.299157 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: No bootstrap required, opened a new log
I20260620 14:21:01.303421 8622 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260620 14:21:01.303793 8622 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:01.303851 8622 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec52afbbfa0d478fbe0fe45ea63d919b, State: Initialized, Role: FOLLOWER
I20260620 14:21:01.304401 8622 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260620 14:21:01.304603 8622 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:01.304692 8622 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:01.304816 8622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:21:01 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:01.306032 8622 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260620 14:21:01.306401 8622 leader_election.cc:302] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec52afbbfa0d478fbe0fe45ea63d919b; no voters:
I20260620 14:21:01.306747 8622 leader_election.cc:288] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:21:01.306843 8627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:01.307116 8627 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 1 LEADER]: Becoming Leader. State: Replica: ec52afbbfa0d478fbe0fe45ea63d919b, State: Running, Role: LEADER
I20260620 14:21:01.307495 8627 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260620 14:21:01.308173 8622 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:01.308804 8629 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } } }
I20260620 14:21:01.309556 8629 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:01.309444 8628 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec52afbbfa0d478fbe0fe45ea63d919b. Latest consensus state: current_term: 1 leader_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } } }
I20260620 14:21:01.312227 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:01.297058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.0.98.62:43089)
0620 14:21:01.297554 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:01.297575 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:01.297618 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:21:01.298364 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:01.298384 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:01.298712 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:01.298990 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:01.299901 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.300906 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.302192 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.302692 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.302810 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:01.305913 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:01.305925 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:01.305962 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:01.305993 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:01.308312 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:01.308865 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:01.308871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:01.308878 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:01.309020 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:01.309377 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:01.309382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:01.309385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:01.309938 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:01.310131 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:01.311160 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0620 14:21:01.311416 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":154,"threads_started":1}
I20260620 14:21:01.312574 8628 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: This master's current role is: LEADER
W20260620 14:21:01.313297 8642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:21:01.313372 8642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:21:01.313445 8638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:01.314260 8638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:01.320029 8638 catalog_manager.cc:1347] Generated new cluster ID: d45b4d6fcaac490e8e69a5a5e6b5ba7d
I20260620 14:21:01.320108 8638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:01.329695 8638 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:21:01.331002 8638 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:01.345616 8638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: Generated new TSK 0
I20260620 14:21:01.346406 8638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:21:01.382372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43089
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:01.514685 8648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:01.514948 8648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:01.514993 8648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:01.519013 8648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:01.519079 8648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:01.519140 8648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:01.524324 8648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:01.525302 8648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:01.526257 8648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:01.536229 8653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:01.536233 8654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:01.536427 8648 server_base.cc:1034] running on GCE node
W20260620 14:21:01.536229 8656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:01.537122 8648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:01.537700 8648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:01.538859 8648 hybrid_clock.cc:648] HybridClock initialized: now 1781965261538846 us; error 24 us; skew 500 ppm
Jun 20 14:21:01 dist-test-slave-87nc krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:01.541538 8648 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:21:01.542347 8648 webserver.cc:460] Webserver started at http://127.0.98.1:34495/ using document root <none> and password file <none>
I20260620 14:21:01.542840 8648 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:01.542913 8648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:01.543071 8648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:01.545111 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "e0ec0eaf64024eeda2155438e9b930ae"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "d7f5b686774a37d01a9a169d14fabee1"
server_key_iv: "0707ab84f62ce8081d7e5e316dd85f93"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.545630 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e0ec0eaf64024eeda2155438e9b930ae"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "d7f5b686774a37d01a9a169d14fabee1"
server_key_iv: "0707ab84f62ce8081d7e5e316dd85f93"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.549403 8648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:21:01.552129 8663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:01.553364 8648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:21:01.553485 8648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "e0ec0eaf64024eeda2155438e9b930ae"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "d7f5b686774a37d01a9a169d14fabee1"
server_key_iv: "0707ab84f62ce8081d7e5e316dd85f93"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.553594 8648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:01.570991 8648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:01.574311 8648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:01.574519 8648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:01.575116 8648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:01.575995 8648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:01.576045 8648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:01.576085 8648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:01.576102 8648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:01.584789 8648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34783
I20260620 14:21:01.584858 8776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34783 every 8 connection(s)
I20260620 14:21:01.585729 8648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:01.589367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 8648
I20260620 14:21:01.589493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:21:01.589768 392 external_mini_cluster.cc:1428] Setting key fddf9cac5d601dfa30b03cb73ed094cb
Jun 20 14:21:01 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:21:01.600555 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:01.587549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35261 (local address 127.0.98.62:43089)
0620 14:21:01.587684 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:01.587692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:01.588465 (+ 773us) server_negotiation.cc:407] Connection header received
0620 14:21:01.590061 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:01.590065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:01.590137 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:01.590236 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:01.591602 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.592193 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.593326 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.593640 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.593741 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:01.596575 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:01.596582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:01.596587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:01.596621 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:01.598349 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:01.598845 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:01.598849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:01.598851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:01.598946 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:01.599335 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:01.599341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:01.599345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:01.599608 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:01.599722 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:01.600323 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 14:21:01.600435 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:21:01.601404 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:01.587882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:35261)
0620 14:21:01.588308 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:01.588350 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:01.589840 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:01.590429 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:01.590436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:01.590934 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:01.591459 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:01.591473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.592317 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.592320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:01.593195 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:01.593204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.593783 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.593790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:01.594032 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:01.594652 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:01.594677 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:01.596404 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:01.598471 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:01.598475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:01.598486 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:01.598739 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:01.599070 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:01.599072 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:01.599074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:01.599224 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:01.599721 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:01.599726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:01.600020 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 14:21:01.600230 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 14:21:01.600483 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:21:01.602670 8777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43089
I20260620 14:21:01.602916 8777 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:01.603428 8777 heartbeater.cc:511] Master 127.0.98.62:43089 requested a full tablet report, sending...
I20260620 14:21:01.605041 8586 ts_manager.cc:194] Registered new tserver with Master: e0ec0eaf64024eeda2155438e9b930ae (127.0.98.1:34783)
I20260620 14:21:01.606667 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:21:01.618893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43089
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:01.727834 8782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:01.728114 8782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:01.728158 8782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:01.732275 8782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:01.732370 8782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:01.732455 8782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:01.737921 8782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:01.738965 8782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:01.740012 8782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:01.749696 8790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:01.749867 8782 server_base.cc:1034] running on GCE node
W20260620 14:21:01.749686 8788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:01.749686 8787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:01.750466 8782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:01.751154 8782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:01.752311 8782 hybrid_clock.cc:648] HybridClock initialized: now 1781965261752295 us; error 27 us; skew 500 ppm
Jun 20 14:21:01 dist-test-slave-87nc krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:01.755107 8782 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:21:01.756093 8782 webserver.cc:460] Webserver started at http://127.0.98.2:35779/ using document root <none> and password file <none>
I20260620 14:21:01.756654 8782 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:01.756742 8782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:01.757014 8782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:01.759217 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "03b808fc80b6497493b8efa6d2d53394"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "a5d5a173cf40159b1905e909c1a0b043"
server_key_iv: "adeedd985604516269bddd21a517992c"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.759851 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "03b808fc80b6497493b8efa6d2d53394"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "a5d5a173cf40159b1905e909c1a0b043"
server_key_iv: "adeedd985604516269bddd21a517992c"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.763604 8782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:21:01.766237 8797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:01.767475 8782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:01.767612 8782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "03b808fc80b6497493b8efa6d2d53394"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "a5d5a173cf40159b1905e909c1a0b043"
server_key_iv: "adeedd985604516269bddd21a517992c"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.767725 8782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:01.779956 8782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:01.782871 8782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:01.783062 8782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:01.783625 8782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:01.784550 8782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:01.784615 8782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:01.784683 8782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:01.784732 8782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:01.793733 8782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41599
I20260620 14:21:01.793752 8910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41599 every 8 connection(s)
I20260620 14:21:01.794695 8782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:21:01 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:01.804965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 8782
I20260620 14:21:01.805114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:21:01.805418 392 external_mini_cluster.cc:1428] Setting key 8fff8b59e56a3fb1332fc323eb8a9a69
I20260620 14:21:01.808670 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:01.796490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56961 (local address 127.0.98.62:43089)
0620 14:21:01.796634 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:01.796638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:01.797464 (+ 826us) server_negotiation.cc:407] Connection header received
0620 14:21:01.798894 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:01.798896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:01.798945 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:01.799020 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:01.800233 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.800743 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.801836 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.802040 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.802112 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:01.804618 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:01.804622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:01.804624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:01.804648 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:01.806302 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:01.806931 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:01.806934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:01.806937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:01.807028 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:01.807372 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:01.807377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:01.807379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:01.807657 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:01.807760 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:01.808304 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 14:21:01.808442 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:21:01.809828 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:01.796876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:56961)
0620 14:21:01.797303 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:01.797344 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:01.798675 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:01.799176 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:01.799182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:01.799530 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:01.800109 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:01.800122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.800860 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.800862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:01.801734 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:01.801742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:01.802159 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:01.802162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:01.802303 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:01.802926 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:01.802942 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:01.804445 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:01.806413 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:01.806419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:01.806436 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:01.806826 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:01.807123 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:01.807125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:01.807126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:01.807259 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:01.807805 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:01.807818 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:01.808055 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 14:21:01.808217 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:21:01.808579 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260620 14:21:01.811061 8911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43089
I20260620 14:21:01.811333 8911 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:01.812008 8911 heartbeater.cc:511] Master 127.0.98.62:43089 requested a full tablet report, sending...
I20260620 14:21:01.813170 8586 ts_manager.cc:194] Registered new tserver with Master: 03b808fc80b6497493b8efa6d2d53394 (127.0.98.2:41599)
I20260620 14:21:01.813795 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56961
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:21:01.832528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43089
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:01.943854 8916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:01.944130 8916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:01.944212 8916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:01.948369 8916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:01.948439 8916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:01.948501 8916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:01.954032 8916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:01.955155 8916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:01.956229 8916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:01.966038 8924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:01.966025 8921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:01.966076 8922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:01.966204 8916 server_base.cc:1034] running on GCE node
I20260620 14:21:01.966828 8916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:01.967506 8916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:01.968686 8916 hybrid_clock.cc:648] HybridClock initialized: now 1781965261968648 us; error 52 us; skew 500 ppm
Jun 20 14:21:01 dist-test-slave-87nc krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:01.971472 8916 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:21:01.972433 8916 webserver.cc:460] Webserver started at http://127.0.98.3:43087/ using document root <none> and password file <none>
I20260620 14:21:01.972996 8916 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:01.973085 8916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:01.973325 8916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:01.975358 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "5fb19fc21d7248a996a3681d1590ac52"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "94503c03a3900dc1c5300801ee1ff6ae"
server_key_iv: "1feac61445ea6c04284c587ee80f73bd"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.975983 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5fb19fc21d7248a996a3681d1590ac52"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "94503c03a3900dc1c5300801ee1ff6ae"
server_key_iv: "1feac61445ea6c04284c587ee80f73bd"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.979624 8916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:21:01.982086 8931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:01.983276 8916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:01.983415 8916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "5fb19fc21d7248a996a3681d1590ac52"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "94503c03a3900dc1c5300801ee1ff6ae"
server_key_iv: "1feac61445ea6c04284c587ee80f73bd"
server_key_version: "encryptionkey@0"
I20260620 14:21:01.983527 8916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:02.002802 8916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:02.006026 8916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:02.006251 8916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:02.006863 8916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:02.007692 8916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:02.007766 8916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.007892 8916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:02.007944 8916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.017366 8916 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46169
I20260620 14:21:02.017385 9044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46169 every 8 connection(s)
I20260620 14:21:02.018265 8916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:21:02.018756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 8916
I20260620 14:21:02.018884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:21:02.019215 392 external_mini_cluster.cc:1428] Setting key be7a162989ba27ebef1a222bc435dc84
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:02.032509 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.020408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50945 (local address 127.0.98.62:43089)
0620 14:21:02.020536 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.020540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.021673 (+ 1133us) server_negotiation.cc:407] Connection header received
0620 14:21:02.023181 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.023184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.023238 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.023313 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.024553 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.025075 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.026186 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.026406 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.026513 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.029057 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:02.029061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:02.029064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:02.029095 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:02.030438 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:02.030941 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:02.030946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:02.030948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:02.031029 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:02.031369 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:02.031374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:02.031376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:02.031566 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:02.031673 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:02.032221 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 14:21:02.032340 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:21:02.033519 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.021053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:50945)
0620 14:21:02.021528 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.021583 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.022932 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.023486 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.023493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.023945 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.024419 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.024431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.025198 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.025208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.026054 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.026075 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.026549 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.026553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.026756 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.027398 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.027413 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.028887 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:02.030551 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:02.030557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:02.030591 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:02.030838 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:02.031126 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:02.031129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:02.031131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:02.031263 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:02.031680 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:02.031688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:02.031985 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 14:21:02.032178 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 14:21:02.032487 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":142,"threads_started":1}
I20260620 14:21:02.034825 9045 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43089
I20260620 14:21:02.035135 9045 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:02.035804 9045 heartbeater.cc:511] Master 127.0.98.62:43089 requested a full tablet report, sending...
I20260620 14:21:02.036993 8586 ts_manager.cc:194] Registered new tserver with Master: 5fb19fc21d7248a996a3681d1590ac52 (127.0.98.3:46169)
I20260620 14:21:02.037578 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50945
I20260620 14:21:02.045148 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:21:02.116849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 8648
I20260620 14:21:02.122357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 8782
I20260620 14:21:02.128026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 8916
I20260620 14:21:02.133046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 8554
I20260620 14:21:02.139284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43089
--webserver_interface=127.0.98.62
--webserver_port=38489
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:02.248271 9054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:02.248546 9054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:02.248589 9054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:02.252744 9054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:02.252808 9054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:02.252830 9054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:02.252846 9054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:02.252900 9054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:02.258195 9054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:02.259352 9054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:02.260452 9054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:02.268779 9060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:02.268786 9062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:02.268833 9059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:02.269164 9054 server_base.cc:1034] running on GCE node
I20260620 14:21:02.269726 9054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:02.270763 9054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:02.271919 9054 hybrid_clock.cc:648] HybridClock initialized: now 1781965262271906 us; error 25 us; skew 500 ppm
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:02.274542 9054 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:02.275446 9054 webserver.cc:460] Webserver started at http://127.0.98.62:38489/ using document root <none> and password file <none>
I20260620 14:21:02.276057 9054 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:02.276153 9054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:02.279903 9054 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 14:21:02.281921 9069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.283082 9054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:02.283202 9054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "ec52afbbfa0d478fbe0fe45ea63d919b"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "81354be05dd438170fbd132fb70222cb"
server_key_iv: "123d69cf359ebfdbc1713567e06a7b1b"
server_key_version: "encryptionkey@0"
I20260620 14:21:02.283718 9054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:02.320927 9054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:02.323735 9054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:02.323951 9054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:02.330775 9054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43089
I20260620 14:21:02.330801 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43089 every 8 connection(s)
I20260620 14:21:02.331828 9054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:02.335837 9122 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:21:02.336076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 9054
I20260620 14:21:02.336556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34783
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43089
--webserver_port=34495
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:21:02.338129 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: Bootstrap starting.
I20260620 14:21:02.354602 9122 log.cc:824] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:02.361068 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:21:02.361439 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: Bootstrap complete.
I20260620 14:21:02.365628 9122 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260620 14:21:02.365996 9122 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec52afbbfa0d478fbe0fe45ea63d919b, State: Initialized, Role: FOLLOWER
I20260620 14:21:02.366492 9122 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260620 14:21:02.366602 9122 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:02.366688 9122 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:02.366782 9122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:21:02.368681 9122 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260620 14:21:02.369006 9122 leader_election.cc:302] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec52afbbfa0d478fbe0fe45ea63d919b; no voters:
I20260620 14:21:02.369304 9122 leader_election.cc:288] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:21:02.369438 9126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:21:02.369669 9126 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [term 2 LEADER]: Becoming Leader. State: Replica: ec52afbbfa0d478fbe0fe45ea63d919b, State: Running, Role: LEADER
I20260620 14:21:02.369982 9126 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260620 14:21:02.370460 9122 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:02.370999 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec52afbbfa0d478fbe0fe45ea63d919b. Latest consensus state: current_term: 2 leader_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } } }
I20260620 14:21:02.371126 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:02.371361 9127 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec52afbbfa0d478fbe0fe45ea63d919b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } } }
I20260620 14:21:02.371465 9127 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:02.372023 9130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:02.375126 9130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:02.375744 9130 catalog_manager.cc:1259] Loaded cluster ID: d45b4d6fcaac490e8e69a5a5e6b5ba7d
I20260620 14:21:02.375829 9130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:02.378125 9130 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:02.378746 9130 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec52afbbfa0d478fbe0fe45ea63d919b: Loaded TSK: 0
I20260620 14:21:02.379980 9130 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:21:02.450596 9124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:02.450865 9124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:02.450908 9124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:02.455338 9124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:02.455411 9124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:02.455472 9124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:02.460896 9124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34495
--tserver_master_addrs=127.0.98.62:43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:02.461956 9124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:02.463003 9124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:02.472571 9151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:02.472587 9149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:02.472775 9124 server_base.cc:1034] running on GCE node
W20260620 14:21:02.472571 9148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:02.473383 9124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:02.474013 9124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:02.475210 9124 hybrid_clock.cc:648] HybridClock initialized: now 1781965262475179 us; error 55 us; skew 500 ppm
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965262, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:02.478154 9124 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:21:02.479104 9124 webserver.cc:460] Webserver started at http://127.0.98.1:34495/ using document root <none> and password file <none>
I20260620 14:21:02.479647 9124 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:02.479750 9124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:02.483762 9124 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:21:02.485903 9158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.487247 9124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:02.487370 9124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "e0ec0eaf64024eeda2155438e9b930ae"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "d7f5b686774a37d01a9a169d14fabee1"
server_key_iv: "0707ab84f62ce8081d7e5e316dd85f93"
server_key_version: "encryptionkey@0"
I20260620 14:21:02.488030 9124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:02.504037 9124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:02.506870 9124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:02.507050 9124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:02.507624 9124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:02.508590 9124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:02.508659 9124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.508729 9124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:02.508783 9124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.517640 9124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34783
I20260620 14:21:02.517659 9271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34783 every 8 connection(s)
I20260620 14:21:02.518584 9124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:02.523138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 9124
I20260620 14:21:02.523692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41599
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43089
--webserver_port=35779
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:02.533210 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.520771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:41467)
0620 14:21:02.521199 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.521248 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.522478 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.523653 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.523662 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.524203 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.524883 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.524904 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.526836 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.526839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.527848 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.527858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.529028 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.529031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.529175 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.529838 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.529854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.532462 (+ 2608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":138,"threads_started":1}
W20260620 14:21:02.533757 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:02.520618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41467 (local address 127.0.98.62:43089)
0620 14:21:02.521007 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.521022 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.521356 (+ 334us) server_negotiation.cc:407] Connection header received
0620 14:21:02.522851 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.522862 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.523175 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.523518 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.525128 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.526689 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.527984 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.528847 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.529069 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.532766 (+ 3697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41467: BlockingRecv error: recv got EOF from 127.0.98.1:41467 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":115,"threads_started":1}
W20260620 14:21:02.533812 9272 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43089 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43089: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:02.539448 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.534268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:50005)
0620 14:21:02.534356 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.534369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.534462 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.534786 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.534788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.534906 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.534988 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.534992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.535981 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.535983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.536492 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.536499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.537032 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.537034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.537116 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.537627 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.537638 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.539359 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:21:02.539628 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:02.534326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50005 (local address 127.0.98.62:43089)
0620 14:21:02.534536 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.534541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.534575 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:21:02.534618 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.534623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.534680 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.534786 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.535092 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.535886 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.536594 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.536940 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.537033 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.539517 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50005: BlockingRecv error: recv got EOF from 127.0.98.1:50005 (error 108)
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":129}
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:02.545011 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.540015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:54301)
0620 14:21:02.540129 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.540145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.540217 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.540414 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.540416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.540524 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.540615 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.540619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.541582 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.541585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.542066 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.542073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.542618 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.542620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.542674 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.543153 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.543164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.544920 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:21:02.545181 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:02.540065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54301 (local address 127.0.98.62:43089)
0620 14:21:02.540162 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.540166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.540209 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:21:02.540280 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.540283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.540322 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.540395 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.540710 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.541455 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.542173 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.542495 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.542592 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.545073 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54301: BlockingRecv error: recv got EOF from 127.0.98.1:54301 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 14:21:02.545418 9272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:21:02.641979 9276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:02.642234 9276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:02.642309 9276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:02.646417 9276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:02.646481 9276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:02.646574 9276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:02.652050 9276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35779
--tserver_master_addrs=127.0.98.62:43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:02.653285 9276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:02.654340 9276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:02.664026 9284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:02.663964 9281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:02.663964 9282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:02.664233 9276 server_base.cc:1034] running on GCE node
I20260620 14:21:02.664654 9276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:02.665313 9276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:02.666500 9276 hybrid_clock.cc:648] HybridClock initialized: now 1781965262666473 us; error 51 us; skew 500 ppm
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965262, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:02.669303 9276 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:21:02.670212 9276 webserver.cc:460] Webserver started at http://127.0.98.2:35779/ using document root <none> and password file <none>
I20260620 14:21:02.670814 9276 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:02.670887 9276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:02.674878 9276 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:02.676864 9291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.678071 9276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:21:02.678190 9276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "03b808fc80b6497493b8efa6d2d53394"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "a5d5a173cf40159b1905e909c1a0b043"
server_key_iv: "adeedd985604516269bddd21a517992c"
server_key_version: "encryptionkey@0"
I20260620 14:21:02.678701 9276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:02.688968 9276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:02.692478 9276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:02.692683 9276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:02.693269 9276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:02.694103 9276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:02.694171 9276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.694235 9276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:02.694290 9276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.703459 9276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41599
I20260620 14:21:02.703478 9404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41599 every 8 connection(s)
I20260620 14:21:02.704434 9276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:02.710604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 9276
I20260620 14:21:02.711021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46169
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43089
--webserver_port=43087
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:21:02.715696 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:02.706221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56889 (local address 127.0.98.62:43089)
0620 14:21:02.706348 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.706351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.707152 (+ 801us) server_negotiation.cc:407] Connection header received
0620 14:21:02.708416 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.708419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.708466 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.708539 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.709614 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.710127 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.711395 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.711823 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.711943 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.715595 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56889: BlockingRecv error: recv got EOF from 127.0.98.2:56889 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:21:02.716596 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.706593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:56889)
0620 14:21:02.707001 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.707040 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.708217 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.708663 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.708669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.709023 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.709483 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.709496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.710235 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.710237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.711264 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.711273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.711956 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.711959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.712093 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.712699 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.712715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.715491 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
W20260620 14:21:02.717090 9405 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43089 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43089: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:02.722559 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.717538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:58099)
0620 14:21:02.717631 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.717646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.717728 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.717990 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.717993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.718153 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.718238 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.718243 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.718963 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.718967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.719530 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.719538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.720059 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.720063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.720153 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.720664 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.720676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.722439 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:21:02.722797 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:02.717642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58099 (local address 127.0.98.62:43089)
0620 14:21:02.717786 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.717790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.717804 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:21:02.717840 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.717842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.717893 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.717977 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.718350 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.718852 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.719647 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.719952 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.720059 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.722560 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58099: BlockingRecv error: recv got EOF from 127.0.98.2:58099 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:02.727706 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.723062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:41365)
0620 14:21:02.723153 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.723168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.723238 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.723518 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.723520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.723677 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.723738 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.723743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.724508 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.724513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.725025 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.725032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.725464 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.725468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.725542 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.726009 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.726021 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.727578 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:21:02.727826 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:02.723155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41365 (local address 127.0.98.62:43089)
0620 14:21:02.723340 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.723344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.723355 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:21:02.723390 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.723392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.723423 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.723504 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.723905 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.724404 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.725137 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.725362 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.725464 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.727678 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41365: BlockingRecv error: recv got EOF from 127.0.98.2:41365 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
W20260620 14:21:02.728035 9405 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:21:02.821921 9408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:02.822206 9408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:02.822250 9408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:02.826412 9408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:02.826471 9408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:02.826529 9408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:02.831679 9408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43087
--tserver_master_addrs=127.0.98.62:43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:02.832751 9408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:02.833739 9408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:02.842957 9417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:02.842971 9414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:02.842980 9415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:02.843250 9408 server_base.cc:1034] running on GCE node
I20260620 14:21:02.843672 9408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:02.844331 9408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:02.845486 9408 hybrid_clock.cc:648] HybridClock initialized: now 1781965262845472 us; error 28 us; skew 500 ppm
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965262, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:02.848181 9408 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:21:02.849043 9408 webserver.cc:460] Webserver started at http://127.0.98.3:43087/ using document root <none> and password file <none>
I20260620 14:21:02.849586 9408 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:02.849676 9408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:02.853431 9408 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:21:02.855316 9424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.856484 9408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 14:21:02.856611 9408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "5fb19fc21d7248a996a3681d1590ac52"
format_stamp: "Formatted at 2026-06-20 14:21:01 on dist-test-slave-87nc"
server_key: "94503c03a3900dc1c5300801ee1ff6ae"
server_key_iv: "1feac61445ea6c04284c587ee80f73bd"
server_key_version: "encryptionkey@0"
I20260620 14:21:02.857132 9408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:02.871891 9408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:02.874709 9408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:02.874893 9408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:02.875440 9408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:02.876328 9408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:02.876395 9408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.876464 9408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:02.876516 9408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:02.885586 9408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46169
I20260620 14:21:02.885605 9537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46169 every 8 connection(s)
I20260620 14:21:02.886617 9408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:21:02.887416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 9408
I20260620 14:21:02.897374 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.888203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.0.98.62:43089)
0620 14:21:02.888367 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.888372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.888577 (+ 205us) server_negotiation.cc:407] Connection header received
0620 14:21:02.888750 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.888756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.888820 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.888914 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.889837 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.890596 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.891530 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.891893 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.892020 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.893474 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:02.893479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:02.893493 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:02.893518 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:02.895254 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:02.895700 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:02.895705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:02.895709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:02.895890 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:02.896193 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:02.896197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:02.896199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:02.896538 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:02.896751 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:02.897000 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:21:02.897243 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":45}
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:21:02.899196 9409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:02.889151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52727 (local address 127.0.98.62:43089)
0620 14:21:02.889257 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.889261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.890108 (+ 847us) server_negotiation.cc:407] Connection header received
0620 14:21:02.891839 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.891842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.891886 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.891975 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.893362 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.893892 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.895028 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.895267 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.895381 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.899083 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52727: BlockingRecv error: recv got EOF from 127.0.98.3:52727 (error 108)
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":44}
I20260620 14:21:02.899881 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.889378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:52727)
0620 14:21:02.889939 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.889995 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.891574 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.892173 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.892186 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.892706 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.893209 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.893222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.894015 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.894020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.894902 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.894911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.895750 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.895753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.895964 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.896511 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.896527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.898976 (+ 2449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":156,"threads_started":1}
W20260620 14:21:02.900308 9538 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43089 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43089: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:02.905184 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.900743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:42497)
0620 14:21:02.900863 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.900877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.900950 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.901116 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.901117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.901236 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.901324 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.901329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.901967 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.901969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.902491 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.902497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.902859 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.902861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.902925 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.903396 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.903408 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.905097 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:21:02.905366 9409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:02.900793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42497 (local address 127.0.98.62:43089)
0620 14:21:02.900914 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.900918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.900938 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:21:02.901011 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.901013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.901047 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.901107 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.901410 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.901883 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.902578 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.902773 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.902845 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.905211 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42497: BlockingRecv error: recv got EOF from 127.0.98.3:42497 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:02 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:02.910454 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:02.905745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:59305)
0620 14:21:02.905870 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:02.905885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:02.905960 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:02.906193 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:02.906196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:02.906338 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:02.906401 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.906406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.907108 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.907113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.907627 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:02.907634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.908164 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.908168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:02.908254 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.908720 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:02.908732 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:02.910364 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:21:02.910658 9409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:02.905803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59305 (local address 127.0.98.62:43089)
0620 14:21:02.905951 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:02.905956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:02.905969 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:21:02.906050 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:02.906052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:02.906097 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:02.906180 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:02.906505 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.906987 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.907752 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:02.908036 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:02.908154 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:02.910485 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59305: BlockingRecv error: recv got EOF from 127.0.98.3:59305 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 14:21:02.910751 9538 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:21:03 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:03 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:03.551815 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:03.546179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:43795)
0620 14:21:03.546528 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:03.546548 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:03.546787 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:03.547091 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:03.547093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:03.547337 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:03.547486 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:03.547495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.548245 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.548247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:03.548824 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:03.548831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.549237 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.549239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:03.549301 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:03.549778 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:03.549789 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:03.551603 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
W20260620 14:21:03.551873 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:03.546336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43795 (local address 127.0.98.62:43089)
0620 14:21:03.546538 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:03.546542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:03.546749 (+ 207us) server_negotiation.cc:407] Connection header received
0620 14:21:03.546890 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:03.546892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:03.546938 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:03.547040 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:03.547591 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.548139 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.548930 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.549143 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.549217 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:03.551719 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43795: BlockingRecv error: recv got EOF from 127.0.98.1:43795 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
Jun 20 14:21:03 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:03 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:03.734258 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:03.728760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:43955)
0620 14:21:03.729064 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:03.729081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:03.729253 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:03.729571 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:03.729575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:03.729781 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:03.729875 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:03.729881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.730627 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.730632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:03.731208 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:03.731216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.731670 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.731674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:03.731759 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:03.732344 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:03.732362 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:03.734132 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
W20260620 14:21:03.734445 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:03.728876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43955 (local address 127.0.98.62:43089)
0620 14:21:03.729030 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:03.729050 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:03.729189 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:21:03.729373 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:03.729377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:03.729426 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:03.729512 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:03.729994 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.730517 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.731335 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.731562 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.731667 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:03.734288 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43955: BlockingRecv error: recv got EOF from 127.0.98.2:43955 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 14:21:03 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:03 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:03.917547 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:03.911589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:53897)
0620 14:21:03.911879 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:03.911893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:03.911991 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:03.912262 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:03.912264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:03.912436 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:03.912529 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:03.912534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.913299 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.913302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:03.913909 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:03.913919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.914506 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.914512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:03.914580 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:03.915178 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:03.915190 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:03.917421 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
W20260620 14:21:03.917677 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:03.911719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53897 (local address 127.0.98.62:43089)
0620 14:21:03.911879 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:03.911882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:03.911941 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:21:03.912104 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:03.912107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:03.912156 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:03.912232 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:03.912651 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.913193 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.914013 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:03.914234 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:03.914322 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:03.917547 (+ 3225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53897: BlockingRecv error: recv got EOF from 127.0.98.3:53897 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 14:21:04 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:04 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:04.557929 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:04.552572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:38021)
0620 14:21:04.552845 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:04.552859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:04.552948 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:04.553195 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:04.553197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:04.553364 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:04.553454 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:04.553459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.554153 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.554156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:04.554737 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:04.554745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.555209 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.555212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:04.555281 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:04.555927 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:04.555940 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:04.557792 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
W20260620 14:21:04.558040 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:04.552708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38021 (local address 127.0.98.62:43089)
0620 14:21:04.552927 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:04.552931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:04.552944 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:21:04.553043 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:04.553045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:04.553087 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:04.553160 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:04.553550 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.554056 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.554834 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.555046 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.555119 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:04.557917 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38021: BlockingRecv error: recv got EOF from 127.0.98.1:38021 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":84,"threads_started":1}
Jun 20 14:21:04 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:04 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:04.740581 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:04.735176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:58067)
0620 14:21:04.735459 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:04.735474 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:04.735607 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:04.735936 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:04.735939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:04.736176 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:04.736286 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:04.736292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.737028 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.737030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:04.737574 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:04.737582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.737984 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.737987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:04.738043 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:04.738502 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:04.738512 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:04.740393 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
W20260620 14:21:04.740622 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:04.735197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58067 (local address 127.0.98.62:43089)
0620 14:21:04.735347 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:04.735351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:04.735578 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:21:04.735704 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:04.735708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:04.735802 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:04.735896 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:04.736420 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.736941 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.737691 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.737896 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.737970 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:04.740521 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58067: BlockingRecv error: recv got EOF from 127.0.98.2:58067 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:21:04 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:04 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:04.924018 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:04.918431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:38211)
0620 14:21:04.918737 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:04.918751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:04.918848 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:04.919088 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:04.919090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:04.919283 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:04.919382 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:04.919387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.920197 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.920199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:04.920765 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:04.920773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.921205 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.921208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:04.921279 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:04.921862 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:04.921878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:04.923883 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
W20260620 14:21:04.924198 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:04.918633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38211 (local address 127.0.98.62:43089)
0620 14:21:04.918777 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:04.918780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:04.918798 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:21:04.918934 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:04.918936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:04.918980 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:04.919051 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:04.919515 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.920097 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.920878 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:04.921100 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:04.921187 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:04.924034 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38211: BlockingRecv error: recv got EOF from 127.0.98.3:38211 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:21:05 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:05 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:05.564530 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:05.558874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:56571)
0620 14:21:05.559197 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:05.559216 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:05.559378 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:05.559650 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:05.559653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:05.559895 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:05.559987 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:05.559993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.560805 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.560807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:05.561363 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:05.561371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.561786 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.561788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:05.561897 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:05.562443 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:05.562455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:05.564389 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":84,"threads_started":1}
W20260620 14:21:05.564672 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:05.558922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56571 (local address 127.0.98.62:43089)
0620 14:21:05.559198 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:05.559204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:05.559317 (+ 113us) server_negotiation.cc:407] Connection header received
0620 14:21:05.559480 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:05.559483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:05.559534 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:05.559629 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:05.560110 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.560639 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.561474 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.561687 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.561772 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:05.564524 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56571: BlockingRecv error: recv got EOF from 127.0.98.1:56571 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
Jun 20 14:21:05 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:05 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:05.746810 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:05.741353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:52479)
0620 14:21:05.741682 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:05.741699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:05.741846 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:05.742125 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:05.742127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:05.742326 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:05.742413 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:05.742419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.743180 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.743187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:05.743811 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:05.743819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.744252 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.744256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:05.744342 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:05.744879 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:05.744891 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:05.746667 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":79,"threads_started":1}
W20260620 14:21:05.746922 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:05.741472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52479 (local address 127.0.98.62:43089)
0620 14:21:05.741661 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:05.741665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:05.741782 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:21:05.741944 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:05.741949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:05.742001 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:05.742088 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:05.742528 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.743072 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.743925 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.744147 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.744250 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:05.746794 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52479: BlockingRecv error: recv got EOF from 127.0.98.2:52479 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 14:21:05 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:05 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:05.930020 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:05.924727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:53493)
0620 14:21:05.925029 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:05.925049 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:05.925146 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:05.925363 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:05.925366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:05.925539 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:05.925631 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:05.925636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.926440 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.926442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:05.926994 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:05.927001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.927434 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.927436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:05.927492 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:05.928054 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:05.928065 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:05.929879 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
W20260620 14:21:05.930107 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:05.924831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53493 (local address 127.0.98.62:43089)
0620 14:21:05.924966 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:05.924969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:05.925117 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:21:05.925210 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:05.925212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:05.925261 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:05.925337 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:05.925763 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.926342 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.927096 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:05.927337 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:05.927423 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:05.930014 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53493: BlockingRecv error: recv got EOF from 127.0.98.3:53493 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 14:21:06 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:06 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:06.574059 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:06.565390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:55337)
0620 14:21:06.565690 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:06.565704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:06.565834 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:06.566072 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:06.566075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:06.566278 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:06.566371 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:06.566376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.567098 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.567101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:06.567651 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:06.567658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.568098 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.568101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:06.568169 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:06.571757 (+ 3588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:06.571810 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:06.573893 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
W20260620 14:21:06.574149 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:06.565509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55337 (local address 127.0.98.62:43089)
0620 14:21:06.565781 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:06.565785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:06.565799 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:21:06.565922 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:06.565924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:06.565964 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:06.566034 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:06.566472 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.566998 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.567753 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.567984 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.568060 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:06.574019 (+ 5959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55337: BlockingRecv error: recv got EOF from 127.0.98.1:55337 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
Jun 20 14:21:06 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:06 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:06.752586 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:06.747518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:54869)
0620 14:21:06.747760 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:06.747796 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:06.747892 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:06.748119 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:06.748121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:06.748290 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:06.748392 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:06.748396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.749099 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.749101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:06.749671 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:06.749678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.750076 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.750078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:06.750134 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:06.750687 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:06.750697 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:06.752475 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
W20260620 14:21:06.752745 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:06.747656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54869 (local address 127.0.98.62:43089)
0620 14:21:06.747852 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:06.747856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:06.747875 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:21:06.747973 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:06.747975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:06.748017 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:06.748083 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:06.748512 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.749012 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.749777 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.749986 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.750062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:06.752597 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54869: BlockingRecv error: recv got EOF from 127.0.98.2:54869 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 14:21:06 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:06 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:06.936538 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:06.931105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:59147)
0620 14:21:06.931367 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:06.931383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:06.931500 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:06.931714 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:06.931715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:06.931906 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:06.932002 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:06.932007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.932739 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.932742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:06.933304 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:06.933312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.933757 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.933761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:06.933831 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:06.934446 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:06.934469 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:06.936421 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
W20260620 14:21:06.936694 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:06.931158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59147 (local address 127.0.98.62:43089)
0620 14:21:06.931323 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:06.931330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:06.931464 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:21:06.931584 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:06.931587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:06.931632 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:06.931700 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:06.932102 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.932613 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.933413 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:06.933622 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:06.933715 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:06.936543 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59147: BlockingRecv error: recv got EOF from 127.0.98.3:59147 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 14:21:07 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:07 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:07.580812 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:07.574915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:57501)
0620 14:21:07.575200 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:07.575219 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:07.575327 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:07.575589 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:07.575592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:07.575813 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:07.575909 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:07.575914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.576632 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.576635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:07.577289 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:07.577297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.577695 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.577699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:07.577767 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:07.578358 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:07.578371 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:07.580644 (+ 2273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
W20260620 14:21:07.580909 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:07.575084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57501 (local address 127.0.98.62:43089)
0620 14:21:07.575326 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:07.575330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:07.575344 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:21:07.575402 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:07.575404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:07.575443 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:07.575519 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:07.576012 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.576529 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.577392 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.577599 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.577671 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:07.580778 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57501: BlockingRecv error: recv got EOF from 127.0.98.1:57501 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
Jun 20 14:21:07 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:07 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:07.759128 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:07.753470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:49079)
0620 14:21:07.753807 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:07.753822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:07.753919 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:07.754184 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:07.754187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:07.754365 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:07.754460 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:07.754465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.755416 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.755420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:07.756281 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:07.756291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.756692 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.756694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:07.756749 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:07.757272 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:07.757284 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:07.759018 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
W20260620 14:21:07.759279 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:07.753693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49079 (local address 127.0.98.62:43089)
0620 14:21:07.753842 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:07.753845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:07.753875 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:21:07.754002 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:07.754004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:07.754046 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:07.754116 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:07.754566 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.755257 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.756385 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.756606 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.756682 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:07.759131 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49079: BlockingRecv error: recv got EOF from 127.0.98.2:49079 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 14:21:07 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:07 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:07.942691 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:07.937431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:50241)
0620 14:21:07.937732 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:07.937747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:07.937857 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:07.938109 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:07.938111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:07.938283 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:07.938378 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:07.938383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.939074 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.939076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:07.939663 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:07.939672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.940142 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.940145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:07.940212 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:07.940785 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:07.940805 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:07.942581 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":130,"threads_started":1}
W20260620 14:21:07.942879 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:07.937515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50241 (local address 127.0.98.62:43089)
0620 14:21:07.937679 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:07.937688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:07.937831 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:21:07.937936 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:07.937939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:07.937986 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:07.938064 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:07.938477 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.938980 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.939812 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:07.940042 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:07.940120 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:07.942718 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50241: BlockingRecv error: recv got EOF from 127.0.98.3:50241 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 14:21:08 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:08 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:08.587088 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:08.581673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:43865)
0620 14:21:08.581952 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:08.581968 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:08.582085 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:08.582340 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:08.582341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:08.582585 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:08.582717 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:08.582724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.583481 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.583484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:08.584076 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:08.584083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.584504 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.584506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:08.584584 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:08.585101 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:08.585112 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:08.586979 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
W20260620 14:21:08.587265 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:08.581715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43865 (local address 127.0.98.62:43089)
0620 14:21:08.581954 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:08.581960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:08.582030 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:21:08.582169 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:08.582171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:08.582218 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:08.582308 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:08.582827 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.583376 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.584177 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.584402 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.584554 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:08.587104 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43865: BlockingRecv error: recv got EOF from 127.0.98.1:43865 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
Jun 20 14:21:08 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:08 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:08.765436 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:08.760083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:52001)
0620 14:21:08.760329 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:08.760342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:08.760438 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:08.760728 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:08.760731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:08.760934 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:08.761027 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:08.761032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.761790 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.761792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:08.762347 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:08.762354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.762818 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.762821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:08.762878 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:08.763361 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:08.763372 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:08.765252 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
W20260620 14:21:08.765509 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:08.760137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52001 (local address 127.0.98.62:43089)
0620 14:21:08.760301 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:08.760304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:08.760406 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:21:08.760556 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:08.760558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:08.760601 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:08.760712 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:08.761137 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.761659 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.762471 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.762715 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.762804 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:08.765398 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52001: BlockingRecv error: recv got EOF from 127.0.98.2:52001 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 14:21:08 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:08 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:08.948995 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:08.943580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:49011)
0620 14:21:08.943865 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:08.943878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:08.944012 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:08.944286 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:08.944289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:08.944491 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:08.944583 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:08.944589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.945364 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.945371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:08.945918 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:08.945926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.946382 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.946385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:08.946469 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:08.946958 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:08.946970 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:08.948825 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
W20260620 14:21:08.949054 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:08.943665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49011 (local address 127.0.98.62:43089)
0620 14:21:08.943830 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:08.943834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:08.943959 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:21:08.944108 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:08.944112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:08.944166 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:08.944249 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:08.944702 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.945236 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.946039 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:08.946275 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:08.946380 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:08.948946 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49011: BlockingRecv error: recv got EOF from 127.0.98.3:49011 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:21:09 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:09 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:09.593462 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:09.587946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:58807)
0620 14:21:09.588293 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:09.588311 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:09.588437 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:09.588736 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:09.588739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:09.588967 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:09.589077 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:09.589085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.590008 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.590011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:09.590560 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:09.590567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.591010 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.591012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:09.591073 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:09.591572 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:09.591584 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:09.593353 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":90,"threads_started":1}
W20260620 14:21:09.593616 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:09.587989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58807 (local address 127.0.98.62:43089)
0620 14:21:09.588277 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:09.588281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:09.588392 (+ 111us) server_negotiation.cc:407] Connection header received
0620 14:21:09.588565 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:09.588568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:09.588621 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:09.588726 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:09.589223 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.589914 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.590653 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.590920 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.590996 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:09.593482 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58807: BlockingRecv error: recv got EOF from 127.0.98.1:58807 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
Jun 20 14:21:09 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:09 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:09.772011 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:09.766448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:50883)
0620 14:21:09.766890 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:09.766905 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:09.767007 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:09.767288 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:09.767292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:09.767547 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:09.767675 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:09.767683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.768449 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.768451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:09.769005 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:09.769012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.769411 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.769413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:09.769470 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:09.769939 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:09.769949 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:09.771881 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":106,"threads_started":1}
W20260620 14:21:09.772159 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:09.766509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50883 (local address 127.0.98.62:43089)
0620 14:21:09.766669 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:09.766672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:09.766969 (+ 297us) server_negotiation.cc:407] Connection header received
0620 14:21:09.767097 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:09.767100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:09.767166 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:09.767236 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:09.767827 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.768358 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.769106 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.769318 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.769396 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:09.772021 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50883: BlockingRecv error: recv got EOF from 127.0.98.2:50883 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 14:21:09 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:09 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:09.956478 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:09.950051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:51811)
0620 14:21:09.950462 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:09.950479 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:09.950617 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:09.951261 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:09.951264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:09.951471 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:09.951631 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:09.951640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.952453 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.952458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:09.953031 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:09.953040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.953536 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.953541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:09.953637 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:09.954214 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:09.954232 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:09.956359 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":108,"threads_started":1}
W20260620 14:21:09.956696 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:09.950147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51811 (local address 127.0.98.62:43089)
0620 14:21:09.950348 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:09.950353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:09.950929 (+ 576us) server_negotiation.cc:407] Connection header received
0620 14:21:09.951006 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:09.951009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:09.951097 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:09.951210 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:09.951800 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.952337 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.953157 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:09.953396 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:09.953506 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:09.956561 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51811: BlockingRecv error: recv got EOF from 127.0.98.3:51811 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:21:10 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:10 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:10.599444 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:10.594285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:35319)
0620 14:21:10.594557 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:10.594572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:10.594657 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:10.594879 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:10.594880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:10.595053 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:10.595145 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:10.595150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.595920 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.595922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:10.596476 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:10.596483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.596930 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.596933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:10.596992 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:10.597535 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:10.597546 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:10.599325 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
W20260620 14:21:10.599654 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:10.594427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35319 (local address 127.0.98.62:43089)
0620 14:21:10.594671 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:10.594674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:10.594689 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:21:10.594722 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:10.594724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:10.594766 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:10.594849 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:10.595249 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.595836 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.596595 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.596830 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.596917 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:10.599470 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35319: BlockingRecv error: recv got EOF from 127.0.98.1:35319 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
Jun 20 14:21:10 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:10 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:10.778846 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:10.773069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:36513)
0620 14:21:10.773374 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:10.773389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:10.773513 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:10.773801 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:10.773804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:10.773987 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:10.774093 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:10.774099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.775212 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.775215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:10.775807 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:10.775816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.776255 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.776257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:10.776317 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:10.776813 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:10.776824 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:10.778682 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":119,"threads_started":1}
W20260620 14:21:10.778887 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:10.773117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36513 (local address 127.0.98.62:43089)
0620 14:21:10.773275 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:10.773279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:10.773528 (+ 249us) server_negotiation.cc:407] Connection header received
0620 14:21:10.773581 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:10.773592 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:10.773668 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:10.773760 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:10.774220 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.775090 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.775909 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.776158 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.776260 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:10.778790 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36513: BlockingRecv error: recv got EOF from 127.0.98.2:36513 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:21:10 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:10 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:10.962723 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:10.957220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:35191)
0620 14:21:10.957488 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:10.957500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:10.957627 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:10.957895 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:10.957898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:10.958099 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:10.958190 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:10.958195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.958990 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.958995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:10.959546 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:10.959554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.960106 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.960110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:10.960210 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:10.960747 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:10.960758 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:10.962592 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
W20260620 14:21:10.962865 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:10.957270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35191 (local address 127.0.98.62:43089)
0620 14:21:10.957430 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:10.957435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:10.957573 (+ 138us) server_negotiation.cc:407] Connection header received
0620 14:21:10.957731 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:10.957734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:10.957786 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:10.957872 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:10.958332 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.958869 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.959662 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:10.959965 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:10.960090 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:10.962721 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35191: BlockingRecv error: recv got EOF from 127.0.98.3:35191 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 14:21:11 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:11 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:11.610191 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:11.600297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:38497)
0620 14:21:11.600562 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:11.600576 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:11.600678 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:11.600976 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:11.600979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:11.601247 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:11.601349 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:11.601355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.602115 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.602121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:11.602679 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:11.602687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.603127 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.603130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:11.603220 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:11.608131 (+ 4911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:11.608143 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:11.610022 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
W20260620 14:21:11.610283 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:11.600353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38497 (local address 127.0.98.62:43089)
0620 14:21:11.600583 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:11.600586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:11.600651 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:21:11.600803 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:11.600806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:11.600856 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:11.600951 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:11.601465 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.602006 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.602796 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.603020 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.603124 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:11.610158 (+ 7034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38497: BlockingRecv error: recv got EOF from 127.0.98.1:38497 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":58,"threads_started":1}
Jun 20 14:21:11 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:11 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:11.785192 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:11.779555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:46127)
0620 14:21:11.779825 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:11.779845 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:11.779940 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:11.780179 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:11.780182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:11.780355 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:11.780444 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:11.780449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.781145 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.781147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:11.781672 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:11.781679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.782050 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.782053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:11.782112 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:11.782739 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:11.782752 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:11.785026 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
W20260620 14:21:11.785259 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:11.779710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46127 (local address 127.0.98.62:43089)
0620 14:21:11.779858 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:11.779861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:11.779894 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:21:11.780038 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:11.780040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:11.780084 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:11.780145 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:11.780534 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.781047 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.781761 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.781963 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.782037 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:11.785144 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46127: BlockingRecv error: recv got EOF from 127.0.98.2:46127 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 14:21:11 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:11 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:11.969251 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:11.963529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:50299)
0620 14:21:11.963851 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:11.963864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:11.963994 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:11.964257 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:11.964260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:11.964464 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:11.964563 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:11.964569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.965364 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.965369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:11.965942 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:11.965950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.966420 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.966424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:11.966509 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:11.967081 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:11.967092 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:11.969081 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":93,"threads_started":1}
W20260620 14:21:11.969380 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:11.963596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50299 (local address 127.0.98.62:43089)
0620 14:21:11.963760 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:11.963766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:11.963937 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:21:11.964096 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:11.964100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:11.964148 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:11.964237 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:11.964683 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.965256 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.966081 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:11.966312 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:11.966419 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:11.969248 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50299: BlockingRecv error: recv got EOF from 127.0.98.3:50299 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:21:12 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:12 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:12.616415 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:12.611052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:37281)
0620 14:21:12.611357 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:12.611371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:12.611456 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:12.611680 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:12.611682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:12.611871 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:12.611963 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:12.611967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.612655 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.612657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:12.613199 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:12.613206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.613589 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.613592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:12.613668 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:12.614338 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:12.614349 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:12.616294 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":114,"threads_started":1}
W20260620 14:21:12.616583 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:12.611187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37281 (local address 127.0.98.62:43089)
0620 14:21:12.611425 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:12.611428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:12.611441 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:21:12.611551 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:12.611554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:12.611596 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:12.611670 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:12.612056 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.612567 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.613290 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.613497 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.613575 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:12.616426 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37281: BlockingRecv error: recv got EOF from 127.0.98.1:37281 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":58,"threads_started":1}
Jun 20 14:21:12 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:12 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:12.791587 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:12.786094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:37939)
0620 14:21:12.786358 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:12.786371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:12.786478 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:12.786790 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:12.786793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:12.786997 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:12.787106 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:12.787111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.788065 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.788068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:12.788641 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:12.788649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.789046 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.789049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:12.789110 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:12.789572 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:12.789583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:12.791427 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
W20260620 14:21:12.791671 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:12.786175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37939 (local address 127.0.98.62:43089)
0620 14:21:12.786331 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:12.786336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:12.786461 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:21:12.786590 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:12.786594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:12.786646 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:12.786748 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:12.787241 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.787978 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.788733 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.788935 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.789031 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:12.791558 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37939: BlockingRecv error: recv got EOF from 127.0.98.2:37939 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 14:21:12 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:12 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:12.975942 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:12.970177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:46621)
0620 14:21:12.970456 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:12.970472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:12.970596 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:12.970829 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:12.970831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:12.971062 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:12.971161 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:12.971166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.972170 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.972176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:12.972892 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:12.972903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.973380 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.973384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:12.973466 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:12.974022 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:12.974046 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:12.975827 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
W20260620 14:21:12.976095 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:12.970331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46621 (local address 127.0.98.62:43089)
0620 14:21:12.970489 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:12.970493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:12.970558 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:21:12.970679 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:12.970681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:12.970726 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:12.970800 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:12.971272 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.972041 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.973022 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:12.973263 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:12.973364 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:12.975940 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46621: BlockingRecv error: recv got EOF from 127.0.98.3:46621 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 14:21:13 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:13 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:13.622378 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:13.617132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:40487)
0620 14:21:13.617417 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:13.617431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:13.617541 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:13.617788 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:13.617791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:13.617980 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:13.618081 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:13.618086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.618814 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.618817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:13.619384 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:13.619393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.619887 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.619889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:13.619950 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:13.620413 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:13.620425 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:13.622252 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
W20260620 14:21:13.622545 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:13.617203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40487 (local address 127.0.98.62:43089)
0620 14:21:13.617424 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:13.617428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:13.617491 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:21:13.617626 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:13.617629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:13.617671 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:13.617746 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:13.618192 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.618698 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.619520 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.619732 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.619875 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:13.622382 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40487: BlockingRecv error: recv got EOF from 127.0.98.1:40487 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
Jun 20 14:21:13 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:13 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:13.797381 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:13.792318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:57945)
0620 14:21:13.792627 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:13.792640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:13.792740 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:13.792977 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:13.792984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:13.793163 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:13.793255 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:13.793260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.793964 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.793967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:13.794575 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:13.794583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.794972 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.794975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:13.795045 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:13.795534 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:13.795547 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:13.797255 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":91,"threads_started":1}
W20260620 14:21:13.797492 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:13.792365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57945 (local address 127.0.98.62:43089)
0620 14:21:13.792515 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:13.792520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:13.792713 (+ 193us) server_negotiation.cc:407] Connection header received
0620 14:21:13.792825 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:13.792827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:13.792872 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:13.792939 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:13.793348 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.793863 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.794672 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.794881 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.794953 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:13.797384 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57945: BlockingRecv error: recv got EOF from 127.0.98.2:57945 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:21:13 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:13 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:13.981904 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:13.976638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:48757)
0620 14:21:13.976916 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:13.976929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:13.977027 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:13.977243 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:13.977245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:13.977424 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:13.977549 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:13.977554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.978257 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.978260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:13.978829 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:13.978837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.979224 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.979227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:13.979284 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:13.979843 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:13.979857 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:13.981724 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":116,"threads_started":1}
W20260620 14:21:13.982095 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:13.976711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48757 (local address 127.0.98.62:43089)
0620 14:21:13.976827 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:13.976832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:13.976989 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:21:13.977100 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:13.977102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:13.977147 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:13.977218 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:13.977640 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.978156 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.978923 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:13.979131 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:13.979210 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:13.981945 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48757: BlockingRecv error: recv got EOF from 127.0.98.3:48757 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 14:21:14 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:14 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:14.630383 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:14.623143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:51435)
0620 14:21:14.623408 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:14.623422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:14.623519 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:14.623857 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:14.623860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:14.624048 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:14.624156 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:14.624162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.625046 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.625052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:14.626152 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:14.626165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.626915 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.626921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:14.627048 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:14.627745 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:14.627789 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:14.630221 (+ 2432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
W20260620 14:21:14.630709 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:14.623234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51435 (local address 127.0.98.62:43089)
0620 14:21:14.623468 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:14.623472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:14.623486 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:21:14.623588 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:14.623590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:14.623630 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:14.623713 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:14.624272 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.624883 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.626308 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.626782 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.626910 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:14.630559 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51435: BlockingRecv error: recv got EOF from 127.0.98.1:51435 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
Jun 20 14:21:14 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:14 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:14.803177 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:14.798128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:55637)
0620 14:21:14.798397 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:14.798410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:14.798518 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:14.798733 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:14.798735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:14.798907 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:14.799002 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:14.799007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.799715 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.799718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:14.800343 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:14.800352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.800753 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.800755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:14.800813 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:14.801273 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:14.801284 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:14.803055 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
W20260620 14:21:14.803337 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:14.798166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55637 (local address 127.0.98.62:43089)
0620 14:21:14.798311 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:14.798315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:14.798485 (+ 170us) server_negotiation.cc:407] Connection header received
0620 14:21:14.798588 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:14.798590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:14.798635 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:14.798704 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:14.799100 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.799618 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.800446 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.800664 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.800738 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:14.803184 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55637: BlockingRecv error: recv got EOF from 127.0.98.2:55637 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:21:14 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:14 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:14.988197 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:14.982721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:43199)
0620 14:21:14.982993 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:14.983006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:14.983110 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:14.983347 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:14.983349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:14.983531 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:14.983632 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:14.983637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.984433 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.984436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:14.984983 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:14.984992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.985411 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.985414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:14.985473 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:14.986066 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:14.986078 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:14.988056 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
W20260620 14:21:14.988345 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:14.982773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43199 (local address 127.0.98.62:43089)
0620 14:21:14.982911 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:14.982914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:14.983080 (+ 166us) server_negotiation.cc:407] Connection header received
0620 14:21:14.983194 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:14.983196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:14.983245 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:14.983308 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:14.983740 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.984337 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.985095 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:14.985311 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:14.985392 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:14.988192 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43199: BlockingRecv error: recv got EOF from 127.0.98.3:43199 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:21:15 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:15 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:15.637135 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:15.631317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:51629)
0620 14:21:15.631574 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:15.631588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:15.631682 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:15.631988 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:15.631990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:15.632157 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:15.632249 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:15.632254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.633033 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.633036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:15.633583 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:15.633591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.634050 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.634052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:15.634110 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:15.634688 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:15.634700 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:15.637016 (+ 2316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
W20260620 14:21:15.637425 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:15.631447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51629 (local address 127.0.98.62:43089)
0620 14:21:15.631693 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:15.631697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:15.631761 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:21:15.631824 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:15.631826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:15.631876 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:15.631959 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:15.632357 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.632918 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.633716 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.633943 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.634035 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:15.637170 (+ 3135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51629: BlockingRecv error: recv got EOF from 127.0.98.1:51629 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
Jun 20 14:21:15 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:15 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:21:15.809427 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:15.804222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40217 (local address 127.0.98.62:43089)
0620 14:21:15.804370 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:15.804373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:15.804525 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:21:15.804647 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:15.804649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:15.804692 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:15.804764 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:15.805225 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.805738 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.806483 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.806700 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.806778 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:15.809327 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40217: BlockingRecv error: recv got EOF from 127.0.98.2:40217 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:21:15.809548 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:15.804042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:40217)
0620 14:21:15.804394 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:15.804410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:15.804562 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:15.804840 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:15.804842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:15.805018 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:15.805113 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:15.805118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.805832 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.805834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:15.806371 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:15.806377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.806798 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.806801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:15.806859 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:15.807381 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:15.807393 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:15.809336 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":156,"threads_started":1}
Jun 20 14:21:15 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:15 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:15.994117 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:15.988988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:37087)
0620 14:21:15.989273 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:15.989287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:15.989385 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:15.989611 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:15.989613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:15.989796 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:15.989889 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:15.989894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.990616 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.990618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:15.991269 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:15.991277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.991707 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.991709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:15.991797 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:15.992281 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:15.992292 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:15.994005 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":105,"threads_started":1}
W20260620 14:21:15.994251 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:15.989031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37087 (local address 127.0.98.62:43089)
0620 14:21:15.989171 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:15.989176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:15.989378 (+ 202us) server_negotiation.cc:407] Connection header received
0620 14:21:15.989452 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:15.989455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:15.989509 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:15.989582 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:15.990003 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.990526 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.991376 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:15.991598 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:15.991685 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:15.994132 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37087: BlockingRecv error: recv got EOF from 127.0.98.3:37087 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:21:16 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:16 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:16.646893 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:16.637930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:37233)
0620 14:21:16.638283 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:16.638305 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:16.638431 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:16.638716 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:16.638719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:16.638968 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:16.639067 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:16.639074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.639966 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.639972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:16.640604 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:16.640613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.641118 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.641122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:16.641215 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:16.644829 (+ 3614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:16.644843 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:16.646765 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":83,"threads_started":1}
W20260620 14:21:16.647085 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:16.638055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37233 (local address 127.0.98.62:43089)
0620 14:21:16.638391 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:16.638398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:16.638466 (+ 68us) server_negotiation.cc:407] Connection header received
0620 14:21:16.638515 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:16.638519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:16.638578 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:16.638688 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:16.639194 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.639837 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.640735 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.640991 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.641117 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:16.646905 (+ 5788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37233: BlockingRecv error: recv got EOF from 127.0.98.1:37233 (error 108)
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":81,"threads_started":1}
Jun 20 14:21:16 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:16 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:16.816205 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:16.810479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:52379)
0620 14:21:16.810788 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:16.810803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:16.810936 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:16.811209 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:16.811211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:16.811487 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:16.811649 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:16.811666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.812647 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.812652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:16.813297 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:16.813306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.813717 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.813719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:16.813780 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:16.814270 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:16.814281 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:16.816086 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":110,"threads_started":1}
W20260620 14:21:16.816330 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:16.810499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52379 (local address 127.0.98.62:43089)
0620 14:21:16.810640 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:16.810644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:16.810880 (+ 236us) server_negotiation.cc:407] Connection header received
0620 14:21:16.811023 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:16.811026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:16.811068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:16.811140 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:16.811799 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.812553 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.813403 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.813613 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.813695 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:16.816218 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52379: BlockingRecv error: recv got EOF from 127.0.98.2:52379 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 14:21:16 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:16 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:17.000448 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:16.994922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.3:39099)
0620 14:21:16.995178 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:16.995192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:16.995289 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:16.995529 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:16.995531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:16.995703 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:16.995828 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:16.995833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.996503 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.996505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:16.997029 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:16.997035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.997494 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.997498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:16.997580 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:16.998230 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:16.998242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:17.000258 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
W20260620 14:21:17.000483 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:16.995039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39099 (local address 127.0.98.62:43089)
0620 14:21:16.995200 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:16.995204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:16.995258 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:21:16.995369 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:16.995371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:16.995429 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:16.995494 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:16.995912 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.996418 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.997117 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:16.997369 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:16.997487 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:17.000378 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39099: BlockingRecv error: recv got EOF from 127.0.98.3:39099 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:21:17 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:17 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:17.654072 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:17.648010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.1:59163)
0620 14:21:17.648311 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:17.648327 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:17.648455 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:17.648701 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:17.648703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:17.648914 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:17.649019 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:17.649026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:17.649763 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:17.649766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:17.650421 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:17.650431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:17.650860 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:17.650864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:17.650946 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:17.651590 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:17.651612 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:17.653951 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
W20260620 14:21:17.654210 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:17.648103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59163 (local address 127.0.98.62:43089)
0620 14:21:17.648349 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:17.648353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:17.648394 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:21:17.648549 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:17.648551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:17.648593 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:17.648685 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:17.649123 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:17.649646 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:17.650529 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:17.650736 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:17.650809 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:17.654064 (+ 3255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59163: BlockingRecv error: recv got EOF from 127.0.98.1:59163 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":102,"threads_started":1}
Jun 20 14:21:17 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:21:17 dist-test-slave-87nc krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:21:17.822666 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:17.817046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43089 (local address 127.0.98.2:45409)
0620 14:21:17.817310 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:17.817323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:17.817412 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:17.817644 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:17.817647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:17.817821 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:17.817915 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:17.817920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:17.818658 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:17.818660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:17.819227 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:17.819235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:17.819646 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:17.819648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:17.819706 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:17.820440 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:17.820452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:17.822523 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
W20260620 14:21:17.822831 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:17.817170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45409 (local address 127.0.98.62:43089)
0620 14:21:17.817307 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:17.817310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:17.817392 (+ 82us) server_negotiation.cc:407] Connection header received
0620 14:21:17.817497 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:17.817499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:17.817541 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:17.817610 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:17.818026 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:17.818564 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:17.819334 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:17.819551 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:17.819628 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:17.822674 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45409: BlockingRecv error: recv got EOF from 127.0.98.2:45409 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:21:17.895079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 9124
I20260620 14:21:17.900594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 9276
I20260620 14:21:17.905843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 9408
I20260620 14:21:17.910838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 9054
2026-06-20T14:21:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17965 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:21:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:21:17Z Disabled control of system clock
I20260620 14:21:17.940917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41799
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:21:18.049450 9604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:18.049727 9604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:18.049762 9604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:18.053746 9604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:18.053814 9604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:18.053829 9604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:18.053846 9604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:18.053862 9604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:18.058768 9604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41799
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:18.059952 9604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:18.061354 9604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:18.070847 9610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:18.070860 9609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:18.070859 9604 server_base.cc:1034] running on GCE node
W20260620 14:21:18.070891 9612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:18.071831 9604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:18.073086 9604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:18.074291 9604 hybrid_clock.cc:648] HybridClock initialized: now 1781965278074275 us; error 39 us; skew 500 ppm
I20260620 14:21:18.076323 9604 webserver.cc:460] Webserver started at http://127.0.98.62:37649/ using document root <none> and password file <none>
I20260620 14:21:18.076959 9604 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:18.077054 9604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:18.077332 9604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:18.079494 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "aa6421bf6abf41a0be17af777a9324c0"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "2461a7c4992dea2b3458589f63bc2821"
server_key_iv: "cf0d9ec9b9b87ae7a40961ba23c7b234"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.080096 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa6421bf6abf41a0be17af777a9324c0"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "2461a7c4992dea2b3458589f63bc2821"
server_key_iv: "cf0d9ec9b9b87ae7a40961ba23c7b234"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.084240 9604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:21:18.086875 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.088318 9604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:21:18.088469 9604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "aa6421bf6abf41a0be17af777a9324c0"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "2461a7c4992dea2b3458589f63bc2821"
server_key_iv: "cf0d9ec9b9b87ae7a40961ba23c7b234"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.088590 9604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:18.109537 9604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:18.110263 9604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:18.110469 9604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:18.117121 9604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41799
I20260620 14:21:18.117169 9670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41799 every 8 connection(s)
I20260620 14:21:18.118047 9604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:18.121125 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:18.126420 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0: Bootstrap starting.
I20260620 14:21:18.126663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 9604
I20260620 14:21:18.126767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:21:18.127089 392 external_mini_cluster.cc:1428] Setting key 0e4b8deeb307c0011e7272b54996020b
I20260620 14:21:18.129144 9671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:18.129884 9671 log.cc:824] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:18.131888 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0: No bootstrap required, opened a new log
I20260620 14:21:18.135414 9671 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6421bf6abf41a0be17af777a9324c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41799 } }
I20260620 14:21:18.135618 9671 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:18.135706 9671 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6421bf6abf41a0be17af777a9324c0, State: Initialized, Role: FOLLOWER
I20260620 14:21:18.136137 9671 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6421bf6abf41a0be17af777a9324c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41799 } }
I20260620 14:21:18.136256 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:18.128431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.0.98.62:41799)
0620 14:21:18.129034 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:18.129047 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:18.129081 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:21:18.129787 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:18.129805 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:18.130264 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:18.130650 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:18.130972 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.132016 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.132945 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.133416 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.133582 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:18.134415 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:18.134420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:18.134445 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:21:18.134801 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:18.134873 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:21:18.134892 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:18.134998 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:18.135112 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:21:18.135357 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20260620 14:21:18.136263 9671 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:18.136543 9671 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:18.136660 9671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:18.137678 9671 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6421bf6abf41a0be17af777a9324c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41799 } }
I20260620 14:21:18.138020 9671 leader_election.cc:302] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa6421bf6abf41a0be17af777a9324c0; no voters:
I20260620 14:21:18.138321 9671 leader_election.cc:288] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:21:18.138422 9676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:18.138659 9676 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [term 1 LEADER]: Becoming Leader. State: Replica: aa6421bf6abf41a0be17af777a9324c0, State: Running, Role: LEADER
I20260620 14:21:18.138985 9676 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6421bf6abf41a0be17af777a9324c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41799 } }
I20260620 14:21:18.139487 9671 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:18.140588 9678 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa6421bf6abf41a0be17af777a9324c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6421bf6abf41a0be17af777a9324c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41799 } } }
I20260620 14:21:18.140715 9678 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:18.141076 9685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:18.141032 9677 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa6421bf6abf41a0be17af777a9324c0. Latest consensus state: current_term: 1 leader_uuid: "aa6421bf6abf41a0be17af777a9324c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6421bf6abf41a0be17af777a9324c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41799 } } }
I20260620 14:21:18.141356 9677 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:18.143723 9685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:18.149510 9685 catalog_manager.cc:1347] Generated new cluster ID: e68107df8c2b41c3ad5eeed570e617e1
I20260620 14:21:18.149580 9685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:18.157771 9685 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:21:18.158963 9685 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:18.175653 9685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa6421bf6abf41a0be17af777a9324c0: Generated new TSK 0
I20260620 14:21:18.176421 9685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:21:18.182382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41799
--builtin_ntp_servers=127.0.98.20:39849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:21:18.297168 9695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:18.297434 9695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:18.297521 9695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:18.302114 9695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:18.302172 9695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:18.302263 9695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:18.307176 9695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:18.308238 9695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:18.309605 9695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:18.319218 9703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:18.319247 9700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:18.319247 9701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:18.319542 9695 server_base.cc:1034] running on GCE node
I20260620 14:21:18.320003 9695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:18.320569 9695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:18.321714 9695 hybrid_clock.cc:648] HybridClock initialized: now 1781965278321696 us; error 31 us; skew 500 ppm
I20260620 14:21:18.323546 9695 webserver.cc:460] Webserver started at http://127.0.98.1:40083/ using document root <none> and password file <none>
I20260620 14:21:18.324167 9695 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:18.324229 9695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:18.324476 9695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:18.326447 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "92c20de525e84272b213aa1ec23c83fe"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "affc9ef78cf1fdcf841facb9ed3f1801"
server_key_iv: "77e3a09fc89020092cdf59bcf6cdb63e"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.326999 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92c20de525e84272b213aa1ec23c83fe"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "affc9ef78cf1fdcf841facb9ed3f1801"
server_key_iv: "77e3a09fc89020092cdf59bcf6cdb63e"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.330543 9695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:21:18.333069 9709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.334306 9695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:21:18.334437 9695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "92c20de525e84272b213aa1ec23c83fe"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "affc9ef78cf1fdcf841facb9ed3f1801"
server_key_iv: "77e3a09fc89020092cdf59bcf6cdb63e"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.334553 9695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:18.345763 9695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:18.346436 9695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:18.346633 9695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:18.347206 9695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:18.348225 9695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:18.348299 9695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.348366 9695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:18.348419 9695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.357173 9695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36425
I20260620 14:21:18.357203 9822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36425 every 8 connection(s)
I20260620 14:21:18.358078 9695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:18.358870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 9695
I20260620 14:21:18.359007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:21:18.359294 392 external_mini_cluster.cc:1428] Setting key 85d6b4dda6dbd7e5ae358693c715322b
I20260620 14:21:18.361474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41799
--builtin_ntp_servers=127.0.98.20:39849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:21:18.369270 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:18.359948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45791 (local address 127.0.98.62:41799)
0620 14:21:18.360084 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:18.360087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:18.361583 (+ 1496us) server_negotiation.cc:407] Connection header received
0620 14:21:18.363550 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:18.363555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:18.363616 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:18.363726 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:18.364729 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.365675 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.366822 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.367261 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.367387 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:18.368658 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:18.368666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:18.368671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:21:18.368691 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:18.368717 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:21:18.368731 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:18.368823 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:18.368978 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:21:18.369048 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:21:18.370013 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:18.360500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41799 (local address 127.0.98.1:45791)
0620 14:21:18.361376 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:18.361446 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:18.363302 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:18.363944 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:18.363951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:18.364038 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:18.364565 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:18.364579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.365817 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.365820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:18.366693 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:18.366701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.367469 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.367473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:18.367620 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:18.368358 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:21:18.368360 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:18.368399 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:21:18.368490 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:18.368819 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:18.368826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:18.368830 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:18.368920 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:21:18.369146 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":151,"threads_started":1}
I20260620 14:21:18.371317 9823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41799
I20260620 14:21:18.371589 9823 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:18.372303 9823 heartbeater.cc:511] Master 127.0.98.62:41799 requested a full tablet report, sending...
I20260620 14:21:18.373759 9635 ts_manager.cc:194] Registered new tserver with Master: 92c20de525e84272b213aa1ec23c83fe (127.0.98.1:36425)
I20260620 14:21:18.374866 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45791
W20260620 14:21:18.475134 9826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:18.475410 9826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:18.475446 9826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:18.479506 9826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:18.479568 9826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:18.479660 9826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:18.484512 9826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:18.485764 9826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:18.487166 9826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:18.497324 9831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:18.497355 9834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:18.497362 9832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:18.497633 9826 server_base.cc:1034] running on GCE node
I20260620 14:21:18.498026 9826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:18.498670 9826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:18.499835 9826 hybrid_clock.cc:648] HybridClock initialized: now 1781965278499814 us; error 31 us; skew 500 ppm
I20260620 14:21:18.501698 9826 webserver.cc:460] Webserver started at http://127.0.98.2:34505/ using document root <none> and password file <none>
I20260620 14:21:18.502285 9826 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:18.502377 9826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:18.502604 9826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:18.504705 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "4e400828046746d69825a45063a71779"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "8892604d08feb88c65ada8b87571b5a6"
server_key_iv: "aec8b32108385a4d34065c8fbb9e99c0"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.505260 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4e400828046746d69825a45063a71779"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "8892604d08feb88c65ada8b87571b5a6"
server_key_iv: "aec8b32108385a4d34065c8fbb9e99c0"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.508976 9826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:21:18.511525 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.512869 9826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:21:18.513000 9826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "4e400828046746d69825a45063a71779"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "8892604d08feb88c65ada8b87571b5a6"
server_key_iv: "aec8b32108385a4d34065c8fbb9e99c0"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.513118 9826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:18.530287 9826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:18.531015 9826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:18.531219 9826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:18.531834 9826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:18.532819 9826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:18.532886 9826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.532965 9826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:18.533004 9826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.542320 9826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45667
I20260620 14:21:18.542346 9953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45667 every 8 connection(s)
I20260620 14:21:18.543231 9826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:18.548491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 9826
I20260620 14:21:18.548627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:21:18.548982 392 external_mini_cluster.cc:1428] Setting key a2b84a6722d492a64f8782925f5b9f8c
I20260620 14:21:18.551052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41799
--builtin_ntp_servers=127.0.98.20:39849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:21:18.552793 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:18.545107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44131 (local address 127.0.98.62:41799)
0620 14:21:18.545234 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:18.545241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:18.546315 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 14:21:18.547630 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:18.547633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:18.547681 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:18.547807 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:18.548765 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.549504 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.550776 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.551003 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.551113 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:18.552291 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:18.552295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:18.552299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:21:18.552316 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:18.552332 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:21:18.552341 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:18.552416 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:18.552584 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:21:18.552654 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:21:18.553653 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:18.545479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41799 (local address 127.0.98.2:44131)
0620 14:21:18.546130 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:18.546170 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:18.547411 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:18.547984 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:18.547991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:18.548074 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:18.548563 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:18.548574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.549631 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.549635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:18.550658 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:18.550667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.551172 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.551177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:18.551333 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:18.551994 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:21:18.551996 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:18.552037 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:21:18.552103 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:18.552430 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:18.552439 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:18.552445 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:21:18.552574 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:21:18.552868 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260620 14:21:18.554859 9954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41799
I20260620 14:21:18.555140 9954 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:18.555881 9954 heartbeater.cc:511] Master 127.0.98.62:41799 requested a full tablet report, sending...
I20260620 14:21:18.557090 9635 ts_manager.cc:194] Registered new tserver with Master: 4e400828046746d69825a45063a71779 (127.0.98.2:45667)
I20260620 14:21:18.557680 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44131
W20260620 14:21:18.664986 9957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:18.665268 9957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:18.665307 9957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:18.669497 9957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:18.669552 9957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:18.669607 9957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:18.674607 9957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:18.675822 9957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:18.677354 9957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:18.687003 9962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:18.686950 9965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:18.686977 9963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:18.687361 9957 server_base.cc:1034] running on GCE node
I20260620 14:21:18.687849 9957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:18.688439 9957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:18.689604 9957 hybrid_clock.cc:648] HybridClock initialized: now 1781965278689591 us; error 31 us; skew 500 ppm
I20260620 14:21:18.691527 9957 webserver.cc:460] Webserver started at http://127.0.98.3:43795/ using document root <none> and password file <none>
I20260620 14:21:18.692171 9957 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:18.692234 9957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:18.692492 9957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:18.694662 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "07aa41896bc846e495bdb26340952ffe"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "e3a2591cbcea4c7e243b5252652e7b07"
server_key_iv: "07fb5668aa7355f8769da4dd3630e2f6"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.695219 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "07aa41896bc846e495bdb26340952ffe"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "e3a2591cbcea4c7e243b5252652e7b07"
server_key_iv: "07fb5668aa7355f8769da4dd3630e2f6"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.699074 9957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:21:18.701670 9971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.702893 9957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:18.703016 9957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "07aa41896bc846e495bdb26340952ffe"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "e3a2591cbcea4c7e243b5252652e7b07"
server_key_iv: "07fb5668aa7355f8769da4dd3630e2f6"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.703125 9957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:18.736635 9957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:18.737365 9957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:18.737601 9957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:18.738251 9957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:18.739298 9957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:18.739352 9957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.739391 9957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:18.739413 9957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.748347 9957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45549
I20260620 14:21:18.748374 10084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45549 every 8 connection(s)
I20260620 14:21:18.749231 9957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:21:18.757901 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:18.750987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49081 (local address 127.0.98.62:41799)
0620 14:21:18.751124 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:18.751127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:18.752216 (+ 1089us) server_negotiation.cc:407] Connection header received
0620 14:21:18.753575 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:18.753578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:18.753623 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:18.753699 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:18.754540 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.755036 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.756190 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.756397 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.756477 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:18.757445 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:18.757448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:18.757450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:21:18.757462 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:18.757479 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:21:18.757487 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:18.757544 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:18.757673 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:21:18.757724 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:21:18.758071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 9957
I20260620 14:21:18.758180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:21:18.758411 392 external_mini_cluster.cc:1428] Setting key c988733696c066540e1178784f04512d
I20260620 14:21:18.758792 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:18.751400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41799 (local address 127.0.98.3:49081)
0620 14:21:18.752088 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:18.752130 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:18.753380 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:18.753851 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:18.753857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:18.753939 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:18.754399 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:18.754411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.755148 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.755151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:18.756086 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:18.756095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:18.756531 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:18.756533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:18.756669 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:18.757200 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:21:18.757202 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:18.757245 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:21:18.757306 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:18.757558 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:18.757564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:18.757568 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:18.757659 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:21:18.757870 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260620 14:21:18.759985 10085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41799
I20260620 14:21:18.760222 10085 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:18.760794 10085 heartbeater.cc:511] Master 127.0.98.62:41799 requested a full tablet report, sending...
I20260620 14:21:18.761749 9635 ts_manager.cc:194] Registered new tserver with Master: 07aa41896bc846e495bdb26340952ffe (127.0.98.3:45549)
I20260620 14:21:18.762315 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49081
I20260620 14:21:18.772478 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:21:18.775350 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:18.774389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.0.98.62:41799)
0620 14:21:18.774510 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:18.774515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:18.774679 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:21:18.774778 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:18.774780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:18.774820 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:18.774892 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:18.775213 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39538: BlockingRecv error: recv got EOF from 127.0.0.1:39538 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:21:18.775601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 9695
I20260620 14:21:18.781011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 9826
I20260620 14:21:18.786478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 9957
I20260620 14:21:18.791601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 9604
2026-06-20T14:21:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (879 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:21:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:21:18Z Disabled control of system clock
I20260620 14:21:18.820178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34131
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:21:18.930549 10096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:18.930830 10096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:18.930866 10096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:18.934947 10096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:18.935005 10096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:18.935020 10096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:18.935037 10096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:18.935052 10096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:18.939873 10096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:18.940997 10096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:18.942387 10096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:18.951159 10102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:18.951239 10096 server_base.cc:1034] running on GCE node
W20260620 14:21:18.951257 10101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:18.951308 10104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:18.952118 10096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:18.953258 10096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:18.954420 10096 hybrid_clock.cc:648] HybridClock initialized: now 1781965278954409 us; error 30 us; skew 500 ppm
I20260620 14:21:18.956430 10096 webserver.cc:460] Webserver started at http://127.0.98.62:35193/ using document root <none> and password file <none>
I20260620 14:21:18.957015 10096 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:18.957078 10096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:18.957288 10096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:18.959300 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "e0b247796ab94839b06c13f50edf3dd7"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "823899c99aad9c4a8e5423d3e93920ac"
server_key_iv: "e6c43fbe9d1554404993c270080f4696"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.959916 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0b247796ab94839b06c13f50edf3dd7"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "823899c99aad9c4a8e5423d3e93920ac"
server_key_iv: "e6c43fbe9d1554404993c270080f4696"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.963526 10096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:21:18.965950 10110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:18.967239 10096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:18.967381 10096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "e0b247796ab94839b06c13f50edf3dd7"
format_stamp: "Formatted at 2026-06-20 14:21:18 on dist-test-slave-87nc"
server_key: "823899c99aad9c4a8e5423d3e93920ac"
server_key_iv: "e6c43fbe9d1554404993c270080f4696"
server_key_version: "encryptionkey@0"
I20260620 14:21:18.967494 10096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:18.972152 10096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:18.972759 10096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:18.972944 10096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:18.979707 10096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34131
I20260620 14:21:18.979713 10162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34131 every 8 connection(s)
I20260620 14:21:18.980644 10096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:18.983688 10163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:18.985814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 10096
I20260620 14:21:18.985900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:21:18.986181 392 external_mini_cluster.cc:1428] Setting key a812b3e3b087b660a47e09f9c3130a86
I20260620 14:21:18.989926 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7: Bootstrap starting.
I20260620 14:21:18.992326 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:18.987637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.0.98.62:34131)
0620 14:21:18.988361 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:18.988379 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:18.988422 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:21:18.989251 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:18.989274 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:18.989711 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:18.990071 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:18.990873 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:18.990878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:18.990892 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:21:18.990907 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:18.990971 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:21:18.990996 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:18.991160 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:18.991230 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:21:18.991462 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":175,"threads_started":1}
I20260620 14:21:18.992554 10163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:18.993382 10163 log.cc:824] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:18.995277 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7: No bootstrap required, opened a new log
I20260620 14:21:18.998588 10163 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b247796ab94839b06c13f50edf3dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34131 } }
I20260620 14:21:18.998818 10163 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:18.998898 10163 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b247796ab94839b06c13f50edf3dd7, State: Initialized, Role: FOLLOWER
I20260620 14:21:18.999372 10163 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b247796ab94839b06c13f50edf3dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34131 } }
I20260620 14:21:18.999519 10163 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:18.999605 10163 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:18.999698 10163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:19.000574 10163 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b247796ab94839b06c13f50edf3dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34131 } }
I20260620 14:21:19.000901 10163 leader_election.cc:302] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0b247796ab94839b06c13f50edf3dd7; no voters:
I20260620 14:21:19.001195 10163 leader_election.cc:288] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:21:19.001302 10168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:19.001582 10168 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [term 1 LEADER]: Becoming Leader. State: Replica: e0b247796ab94839b06c13f50edf3dd7, State: Running, Role: LEADER
I20260620 14:21:19.001957 10168 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b247796ab94839b06c13f50edf3dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34131 } }
I20260620 14:21:19.002401 10163 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:19.003527 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0b247796ab94839b06c13f50edf3dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b247796ab94839b06c13f50edf3dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34131 } } }
I20260620 14:21:19.003676 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:19.003969 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b247796ab94839b06c13f50edf3dd7. Latest consensus state: current_term: 1 leader_uuid: "e0b247796ab94839b06c13f50edf3dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b247796ab94839b06c13f50edf3dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34131 } } }
I20260620 14:21:19.004144 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7 [sys.catalog]: This master's current role is: LEADER
W20260620 14:21:19.006984 10183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:21:19.007084 10183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:21:19.007174 10177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:19.007936 10177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:19.013243 10177 catalog_manager.cc:1347] Generated new cluster ID: 7072e0d649ab480b822e3a64d7a8270d
I20260620 14:21:19.013307 10177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:19.027307 10177 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:21:19.027659 10177 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:19.033887 10177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0b247796ab94839b06c13f50edf3dd7: Generated new TSK 0
I20260620 14:21:19.034729 10177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:21:19.039908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34131
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:21:19.151413 10187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:19.151702 10187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:19.151824 10187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:19.155948 10187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:19.156028 10187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:19.156118 10187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:19.160972 10187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:19.162084 10187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:19.163457 10187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:19.173302 10195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:19.173276 10192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:19.173318 10193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:19.173290 10187 server_base.cc:1034] running on GCE node
I20260620 14:21:19.173899 10187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:19.174530 10187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:19.175716 10187 hybrid_clock.cc:648] HybridClock initialized: now 1781965279175706 us; error 32 us; skew 500 ppm
I20260620 14:21:19.177524 10187 webserver.cc:460] Webserver started at http://127.0.98.1:37359/ using document root <none> and password file <none>
I20260620 14:21:19.178119 10187 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:19.178229 10187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:19.178452 10187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:19.180505 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "3a7ac9f833f54640b48e1011b0d62292"
format_stamp: "Formatted at 2026-06-20 14:21:19 on dist-test-slave-87nc"
server_key: "9021e1169fca7eb9d60f716f2e4b2223"
server_key_iv: "bfa9a584f07c4753fc96b91fda47c6f9"
server_key_version: "encryptionkey@0"
I20260620 14:21:19.181057 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3a7ac9f833f54640b48e1011b0d62292"
format_stamp: "Formatted at 2026-06-20 14:21:19 on dist-test-slave-87nc"
server_key: "9021e1169fca7eb9d60f716f2e4b2223"
server_key_iv: "bfa9a584f07c4753fc96b91fda47c6f9"
server_key_version: "encryptionkey@0"
I20260620 14:21:19.184609 10187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:21:19.186883 10201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:19.188185 10187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:21:19.188318 10187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "3a7ac9f833f54640b48e1011b0d62292"
format_stamp: "Formatted at 2026-06-20 14:21:19 on dist-test-slave-87nc"
server_key: "9021e1169fca7eb9d60f716f2e4b2223"
server_key_iv: "bfa9a584f07c4753fc96b91fda47c6f9"
server_key_version: "encryptionkey@0"
I20260620 14:21:19.188426 10187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:19.192950 10187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:19.193545 10187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:19.193735 10187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:19.194276 10187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:19.195152 10187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:19.195222 10187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:19.195292 10187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:19.195334 10187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:19.204099 10187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36545
I20260620 14:21:19.204104 10314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36545 every 8 connection(s)
I20260620 14:21:19.205021 10187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:19.206180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 10187
I20260620 14:21:19.206291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:21:19.206638 392 external_mini_cluster.cc:1428] Setting key ba0bcb3cb5e05493fc255b4504610809
I20260620 14:21:19.208720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34131
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:21:19.211740 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:19.207248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47045 (local address 127.0.98.62:34131)
0620 14:21:19.207365 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:19.207368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:19.208389 (+ 1021us) server_negotiation.cc:407] Connection header received
0620 14:21:19.209872 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:19.209879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:19.209920 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:19.210002 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:19.211230 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:19.211232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:19.211234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:21:19.211245 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:19.211286 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:21:19.211299 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:19.211405 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:19.211555 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:21:19.211598 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:21:19.212639 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:19.207427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34131 (local address 127.0.98.1:47045)
0620 14:21:19.208243 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:19.208295 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:19.209613 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:19.210209 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:19.210219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:19.210336 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:19.210946 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:21:19.210948 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:19.211001 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:21:19.211080 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:19.211396 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:19.211404 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:19.211411 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:21:19.211595 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 14:21:19.211851 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260620 14:21:19.213945 10315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34131
I20260620 14:21:19.214169 10315 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:19.214550 10315 heartbeater.cc:511] Master 127.0.98.62:34131 requested a full tablet report, sending...
I20260620 14:21:19.216217 10127 ts_manager.cc:194] Registered new tserver with Master: 3a7ac9f833f54640b48e1011b0d62292 (127.0.98.1:36545)
W20260620 14:21:19.320638 10318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:19.320925 10318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:19.320967 10318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:19.324916 10318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:19.324970 10318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:19.325028 10318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:19.329767 10318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:19.330830 10318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:19.332229 10318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:19.341676 10324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:19.341717 10326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:19.341717 10323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:19.341881 10318 server_base.cc:1034] running on GCE node
I20260620 14:21:19.342466 10318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:19.343123 10318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:19.344318 10318 hybrid_clock.cc:648] HybridClock initialized: now 1781965279344279 us; error 51 us; skew 500 ppm
I20260620 14:21:19.346231 10318 webserver.cc:460] Webserver started at http://127.0.98.2:44293/ using document root <none> and password file <none>
I20260620 14:21:19.346832 10318 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:19.346925 10318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:19.347162 10318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:19.349416 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "102f2ca0eb134fefa054ed26d8ab3a79"
format_stamp: "Formatted at 2026-06-20 14:21:19 on dist-test-slave-87nc"
server_key: "31c34043a9379537ea1351929a95d4c1"
server_key_iv: "91060aeb4ad51dd41a5e45dad87937d2"
server_key_version: "encryptionkey@0"
I20260620 14:21:19.350020 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "102f2ca0eb134fefa054ed26d8ab3a79"
format_stamp: "Formatted at 2026-06-20 14:21:19 on dist-test-slave-87nc"
server_key: "31c34043a9379537ea1351929a95d4c1"
server_key_iv: "91060aeb4ad51dd41a5e45dad87937d2"
server_key_version: "encryptionkey@0"
I20260620 14:21:19.353945 10318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:21:19.356719 10332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:19.358080 10318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:21:19.358237 10318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "102f2ca0eb134fefa054ed26d8ab3a79"
format_stamp: "Formatted at 2026-06-20 14:21:19 on dist-test-slave-87nc"
server_key: "31c34043a9379537ea1351929a95d4c1"
server_key_iv: "91060aeb4ad51dd41a5e45dad87937d2"
server_key_version: "encryptionkey@0"
I20260620 14:21:19.358362 10318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:19.363135 10318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:19.363741 10318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:19.363965 10318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:19.364521 10318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:19.365432 10318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:19.365499 10318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:19.365567 10318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:19.365614 10318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:19.375053 10318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39295
I20260620 14:21:19.375072 10445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39295 every 8 connection(s)
I20260620 14:21:19.376087 10318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:19.382393 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:19.378003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53165 (local address 127.0.98.62:34131)
0620 14:21:19.378142 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:19.378146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:19.379269 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 14:21:19.380668 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:19.380671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:19.380716 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:19.380798 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:19.381912 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:19.381916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:19.381919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:21:19.381931 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:19.381952 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:21:19.381961 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:19.382068 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:19.382215 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:21:19.382259 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:21:19.383275 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:19.378446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34131 (local address 127.0.98.2:53165)
0620 14:21:19.379109 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:19.379158 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:19.380445 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:19.380956 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:19.380963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:19.381052 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:19.381657 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:21:19.381658 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:19.381700 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:21:19.381768 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:19.382051 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:19.382059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:19.382064 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:21:19.382250 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:21:19.382485 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260620 14:21:19.384428 10446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34131
I20260620 14:21:19.384653 10446 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:19.385044 10446 heartbeater.cc:511] Master 127.0.98.62:34131 requested a full tablet report, sending...
I20260620 14:21:19.385361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 10318
I20260620 14:21:19.385495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:21:19.385859 392 external_mini_cluster.cc:1428] Setting key 1be96a69831dbf1dc0397bb8b0bffeeb
I20260620 14:21:19.386108 10127 ts_manager.cc:194] Registered new tserver with Master: 102f2ca0eb134fefa054ed26d8ab3a79 (127.0.98.2:39295)
I20260620 14:21:19.387946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34131
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:21:19.499366 10449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:19.499645 10449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:19.499686 10449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:19.503927 10449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:19.503983 10449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:19.504074 10449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:19.509023 10449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:19.510038 10449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:19.511389 10449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:19.521234 10455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:19.521258 10454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:19.521311 10449 server_base.cc:1034] running on GCE node
W20260620 14:21:19.521242 10457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:19.521790 10449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:19.522365 10449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:19.523521 10449 hybrid_clock.cc:648] HybridClock initialized: now 1781965279523495 us; error 37 us; skew 500 ppm
I20260620 14:21:19.525429 10449 webserver.cc:460] Webserver started at http://127.0.98.3:35169/ using document root <none> and password file <none>
I20260620 14:21:19.526010 10449 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:19.526072 10449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:19.526332 10449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:19.528450 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "50a4a54ad54f41f5a21d8f004138080e"
format_stamp: "Formatted at 2026-06-20 14:21:19 on dist-test-slave-87nc"
server_key: "053c5606eb3677a7cedb173d8db81700"
server_key_iv: "2c26ce0f56f3253a8d44f394601289ca"
server_key_version: "encryptionkey@0"
I20260620 14:21:19.529007 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "50a4a54ad54f41f5a21d8f004138080e"
format_stamp: "Formatted at 2026-06-20 14:21:19 on dist-test-slave-87nc"
server_key: "053c5606eb3677a7cedb173d8db81700"
server_key_iv: "2c26ce0f56f3253a8d44f394601289ca"
server_key_version: "encryptionkey@0"
I20260620 14:21:19.533105 10449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 14:21:19.535490 10463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:19.536773 10449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:21:19.536902 10449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "50a4a54ad54f41f5a21d8f004138080e"
format_stamp: "Formatted at 2026-06-20 14:21:19 on dist-test-slave-87nc"
server_key: "053c5606eb3677a7cedb173d8db81700"
server_key_iv: "2c26ce0f56f3253a8d44f394601289ca"
server_key_version: "encryptionkey@0"
I20260620 14:21:19.537029 10449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:19.541633 10449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:19.542248 10449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:19.542428 10449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:19.542963 10449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:19.544103 10449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:19.544171 10449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:19.544232 10449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:19.544261 10449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:19.553171 10449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42007
I20260620 14:21:19.553193 10576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42007 every 8 connection(s)
I20260620 14:21:19.554183 10449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:21:19.560391 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:19.556079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36863 (local address 127.0.98.62:34131)
0620 14:21:19.556213 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:19.556217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:19.557396 (+ 1179us) server_negotiation.cc:407] Connection header received
0620 14:21:19.558631 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:19.558634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:19.558672 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:19.558751 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:19.559985 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:19.559988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:19.559990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:21:19.560002 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:19.560022 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:21:19.560030 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:19.560112 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:19.560245 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:21:19.560281 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:21:19.561327 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:19.556555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34131 (local address 127.0.98.3:36863)
0620 14:21:19.557227 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:19.557277 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:19.558442 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:19.558914 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:19.558921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:19.559010 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:19.559622 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:21:19.559624 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:19.559667 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:21:19.559737 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:19.560115 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:19.560121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:19.560125 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:19.560260 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:21:19.560458 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260620 14:21:19.562577 10577 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34131
I20260620 14:21:19.562815 10577 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:19.563231 10577 heartbeater.cc:511] Master 127.0.98.62:34131 requested a full tablet report, sending...
I20260620 14:21:19.563957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 10449
I20260620 14:21:19.564083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:21:19.564378 392 external_mini_cluster.cc:1428] Setting key 2f167c2cc11c5d8de4f13d17a7923d2a
I20260620 14:21:19.564393 10126 ts_manager.cc:194] Registered new tserver with Master: 50a4a54ad54f41f5a21d8f004138080e (127.0.98.3:42007)
I20260620 14:21:19.566318 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:21:19.569109 10166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:19.568251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.0.98.62:34131)
0620 14:21:19.568370 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:19.568375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:19.568557 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:21:19.568664 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:19.568666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:19.568696 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:19.568762 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:19.568991 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35748: BlockingRecv error: recv got EOF from 127.0.0.1:35748 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:21:19.569257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 10187
I20260620 14:21:19.575584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 10318
I20260620 14:21:19.581156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 10449
I20260620 14:21:19.586422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 10096
2026-06-20T14:21:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (794 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:21:19 dist-test-slave-87nc krb5kdc[10587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:21:19 dist-test-slave-87nc krb5kdc[10587](info): set up 2 sockets
Jun 20 14:21:19 dist-test-slave-87nc krb5kdc[10587](info): commencing operation
krb5kdc: starting...
W20260620 14:21:20.634599 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:21:20 dist-test-slave-87nc krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:21:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:21:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:21:20.756870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34269
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:20.864259 10600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:20.864531 10600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:20.864571 10600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:20.868566 10600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:20.868618 10600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:20.868636 10600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:20.868654 10600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:20.868721 10600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:20.874053 10600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:20.875159 10600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:20.876211 10600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:21:20.884574 10600 server_base.cc:1034] running on GCE node
W20260620 14:21:20.884539 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:20.884539 10605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:20.884567 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:20.885351 10600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:20.886355 10600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:20.887557 10600 hybrid_clock.cc:648] HybridClock initialized: now 1781965280887521 us; error 50 us; skew 500 ppm
Jun 20 14:21:20 dist-test-slave-87nc krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:20.890183 10600 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:20.891117 10600 webserver.cc:460] Webserver started at http://127.0.98.62:46541/ using document root <none> and password file <none>
I20260620 14:21:20.891661 10600 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:20.891760 10600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:20.892030 10600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:20.894008 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "eced47264d0740b380773bb9a055eb1a"
format_stamp: "Formatted at 2026-06-20 14:21:20 on dist-test-slave-87nc"
server_key: "3cd1f8f70f78da1e9a452fd0ad2bb760"
server_key_iv: "c96fcecdf816595f93cd6d113597386b"
server_key_version: "encryptionkey@0"
I20260620 14:21:20.894546 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "eced47264d0740b380773bb9a055eb1a"
format_stamp: "Formatted at 2026-06-20 14:21:20 on dist-test-slave-87nc"
server_key: "3cd1f8f70f78da1e9a452fd0ad2bb760"
server_key_iv: "c96fcecdf816595f93cd6d113597386b"
server_key_version: "encryptionkey@0"
I20260620 14:21:20.898098 10600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:21:20.900341 10615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:20.901441 10600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:21:20.901573 10600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "eced47264d0740b380773bb9a055eb1a"
format_stamp: "Formatted at 2026-06-20 14:21:20 on dist-test-slave-87nc"
server_key: "3cd1f8f70f78da1e9a452fd0ad2bb760"
server_key_iv: "c96fcecdf816595f93cd6d113597386b"
server_key_version: "encryptionkey@0"
I20260620 14:21:20.901674 10600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:20.917625 10600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:20.920343 10600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:20.920545 10600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:20.927241 10600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34269
I20260620 14:21:20.927266 10667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34269 every 8 connection(s)
I20260620 14:21:20.928162 10600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:20.930735 10668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:20.932539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 10600
I20260620 14:21:20.932663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:21:20.932955 392 external_mini_cluster.cc:1428] Setting key 16fbd2dd2552f034b06f05fa87019d4a
I20260620 14:21:20.937100 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a: Bootstrap starting.
I20260620 14:21:20.939270 10668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:20.940040 10668 log.cc:824] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:21:20 dist-test-slave-87nc krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:20.941943 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a: No bootstrap required, opened a new log
I20260620 14:21:20.945503 10668 raft_consensus.cc:348] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced47264d0740b380773bb9a055eb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 14:21:20.945756 10668 raft_consensus.cc:374] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:20.945815 10668 raft_consensus.cc:729] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eced47264d0740b380773bb9a055eb1a, State: Initialized, Role: FOLLOWER
I20260620 14:21:20.946332 10668 consensus_queue.cc:260] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced47264d0740b380773bb9a055eb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 14:21:20.946548 10668 raft_consensus.cc:388] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:20.946645 10668 raft_consensus.cc:482] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:20.946748 10668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:20.947310 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:20.934405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.0.98.62:34269)
0620 14:21:20.934825 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:20.934840 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:20.934874 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:21:20.935497 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:20.935513 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:20.935891 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:20.936220 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:20.937180 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:20.938282 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:20.939197 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:20.939733 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:20.939949 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:20.942565 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:20.942570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:20.942583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:20.942609 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:20.944397 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:20.944850 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:20.944856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:20.944868 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:20.944977 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:20.945281 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:20.945284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:20.945285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:20.945677 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:20.945895 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:20.946162 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:21:20.946448 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260620 14:21:20.947888 10668 raft_consensus.cc:504] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced47264d0740b380773bb9a055eb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 14:21:20.948235 10668 leader_election.cc:302] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eced47264d0740b380773bb9a055eb1a; no voters:
I20260620 14:21:20.948597 10668 leader_election.cc:288] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:21:20.948705 10673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:20.948902 10673 raft_consensus.cc:686] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [term 1 LEADER]: Becoming Leader. State: Replica: eced47264d0740b380773bb9a055eb1a, State: Running, Role: LEADER
I20260620 14:21:20.949196 10673 consensus_queue.cc:237] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced47264d0740b380773bb9a055eb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 14:21:20.950002 10668 sys_catalog.cc:564] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:20.950349 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eced47264d0740b380773bb9a055eb1a. Latest consensus state: current_term: 1 leader_uuid: "eced47264d0740b380773bb9a055eb1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced47264d0740b380773bb9a055eb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } } }
I20260620 14:21:20.950479 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:20.950806 10674 sys_catalog.cc:455] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eced47264d0740b380773bb9a055eb1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced47264d0740b380773bb9a055eb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } } }
I20260620 14:21:20.950893 10677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:20.950908 10674 sys_catalog.cc:458] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:20.953960 10677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:20.960075 10677 catalog_manager.cc:1347] Generated new cluster ID: 93f1ed6f2aee4d3ebc39f29d4070689b
I20260620 14:21:20.960161 10677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:20.969699 10677 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:21:20.970556 10677 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:20.974220 10677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a: Generated new TSK 0
I20260620 14:21:20.974915 10677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:21:21.006742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:21.132272 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:21.132619 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:21.132669 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:21.136677 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:21.136736 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:21.136793 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:21.141988 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:21.143110 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:21.144177 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:21.154090 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:21.154163 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:21.154107 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:21.154526 10694 server_base.cc:1034] running on GCE node
I20260620 14:21:21.154939 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:21.155619 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:21.156811 10694 hybrid_clock.cc:648] HybridClock initialized: now 1781965281156783 us; error 52 us; skew 500 ppm
Jun 20 14:21:21 dist-test-slave-87nc krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:21.159646 10694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:21:21.160544 10694 webserver.cc:460] Webserver started at http://127.0.98.1:37191/ using document root <none> and password file <none>
I20260620 14:21:21.161056 10694 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:21.161124 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:21.161310 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:21.163311 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "1a346359e1254b888cf1fe0254466e7c"
format_stamp: "Formatted at 2026-06-20 14:21:21 on dist-test-slave-87nc"
server_key: "6f0fb23a1603a81d31cbb1618f665653"
server_key_iv: "09f46622e6cf75abb46aa88f2d0df398"
server_key_version: "encryptionkey@0"
I20260620 14:21:21.163888 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1a346359e1254b888cf1fe0254466e7c"
format_stamp: "Formatted at 2026-06-20 14:21:21 on dist-test-slave-87nc"
server_key: "6f0fb23a1603a81d31cbb1618f665653"
server_key_iv: "09f46622e6cf75abb46aa88f2d0df398"
server_key_version: "encryptionkey@0"
I20260620 14:21:21.167459 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:21:21.169901 10709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:21.171206 10694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:21.171309 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "1a346359e1254b888cf1fe0254466e7c"
format_stamp: "Formatted at 2026-06-20 14:21:21 on dist-test-slave-87nc"
server_key: "6f0fb23a1603a81d31cbb1618f665653"
server_key_iv: "09f46622e6cf75abb46aa88f2d0df398"
server_key_version: "encryptionkey@0"
I20260620 14:21:21.171394 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:21.191577 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:21.195346 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:21.195525 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:21.196187 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:21.197000 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:21.197048 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:21.197088 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:21.197103 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:21.206248 10694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42927
I20260620 14:21:21.206341 10822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42927 every 8 connection(s)
I20260620 14:21:21.207242 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:21.213986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 10694
I20260620 14:21:21.214108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:21:21.214433 392 external_mini_cluster.cc:1428] Setting key 452598103c2982371be19b4ba54c7c79
Jun 20 14:21:21 dist-test-slave-87nc krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:21.222195 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.209233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46309 (local address 127.0.98.62:34269)
0620 14:21:21.209399 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.209403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.210239 (+ 836us) server_negotiation.cc:407] Connection header received
0620 14:21:21.211565 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.211569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.211626 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.211733 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:21.212895 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.213436 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.215063 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.215384 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.215493 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.218378 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:21.218385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:21.218389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:21.218432 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:21.220105 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:21.220655 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:21.220660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:21.220662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:21.220746 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:21.221082 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:21.221087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:21.221088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:21.221276 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:21.221382 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.221895 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.222013 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":77}
I20260620 14:21:21.223121 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.209605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34269 (local address 127.0.98.1:46309)
0620 14:21:21.210059 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:21.210110 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:21.211343 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:21.211931 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:21.211938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:21.212291 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:21.212749 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.212762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.213560 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.213564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.214919 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.214928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.215555 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.215558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.215803 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.216469 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:21.216491 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:21.218199 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:21.220245 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:21.220251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:21.220269 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:21.220528 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:21.220843 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:21.220847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:21.220849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:21.220989 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:21.221393 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:21.221401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:21.221665 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 14:21:21.221856 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:21:21.222195 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:21:21.224480 10823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34269
I20260620 14:21:21.224738 10823 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:21.225309 10823 heartbeater.cc:511] Master 127.0.98.62:34269 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:21:21.226884 10632 ts_manager.cc:194] Registered new tserver with Master: 1a346359e1254b888cf1fe0254466e7c (127.0.98.1:42927)
I20260620 14:21:21.228070 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:21:21.242498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:21.354231 10828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:21.354523 10828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:21.354615 10828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:21.358764 10828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:21.358839 10828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:21.358960 10828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:21.364894 10828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:21.366037 10828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:21.367089 10828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:21.376791 10836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:21.376797 10833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:21.376932 10828 server_base.cc:1034] running on GCE node
W20260620 14:21:21.376804 10834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:21.377521 10828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:21.378152 10828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:21.379325 10828 hybrid_clock.cc:648] HybridClock initialized: now 1781965281379291 us; error 46 us; skew 500 ppm
Jun 20 14:21:21 dist-test-slave-87nc krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:21.382252 10828 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:21:21.383213 10828 webserver.cc:460] Webserver started at http://127.0.98.2:33199/ using document root <none> and password file <none>
I20260620 14:21:21.383826 10828 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:21.383889 10828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:21.384114 10828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:21.386153 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "b97c2b61e3d940198d08e7421064d348"
format_stamp: "Formatted at 2026-06-20 14:21:21 on dist-test-slave-87nc"
server_key: "23879265bab8bcc2d97c9e48c9419add"
server_key_iv: "a82f2a6aa7ff9358ebca6de0b6292fda"
server_key_version: "encryptionkey@0"
I20260620 14:21:21.386713 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b97c2b61e3d940198d08e7421064d348"
format_stamp: "Formatted at 2026-06-20 14:21:21 on dist-test-slave-87nc"
server_key: "23879265bab8bcc2d97c9e48c9419add"
server_key_iv: "a82f2a6aa7ff9358ebca6de0b6292fda"
server_key_version: "encryptionkey@0"
I20260620 14:21:21.390312 10828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:21:21.393021 10843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:21.394322 10828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:21:21.394464 10828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "b97c2b61e3d940198d08e7421064d348"
format_stamp: "Formatted at 2026-06-20 14:21:21 on dist-test-slave-87nc"
server_key: "23879265bab8bcc2d97c9e48c9419add"
server_key_iv: "a82f2a6aa7ff9358ebca6de0b6292fda"
server_key_version: "encryptionkey@0"
I20260620 14:21:21.394593 10828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:21.435261 10828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:21.438321 10828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:21.438550 10828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:21.439170 10828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:21.440045 10828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:21.440091 10828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:21.440160 10828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:21.440193 10828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:21.449509 10828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38305
I20260620 14:21:21.449549 10956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38305 every 8 connection(s)
I20260620 14:21:21.450618 10828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:21.460119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 10828
I20260620 14:21:21.460242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
Jun 20 14:21:21 dist-test-slave-87nc krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:21.460557 392 external_mini_cluster.cc:1428] Setting key 09adb84f909296e8f356b462e36bb0f7
I20260620 14:21:21.464720 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.452614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46633 (local address 127.0.98.62:34269)
0620 14:21:21.452780 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.452785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.453771 (+ 986us) server_negotiation.cc:407] Connection header received
0620 14:21:21.455054 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.455057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.455105 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.455189 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:21.456506 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.457026 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.458171 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.458411 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.458491 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.461153 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:21.461156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:21.461159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:21.461183 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:21.462601 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:21.463123 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:21.463130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:21.463132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:21.463213 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:21.463560 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:21.463564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:21.463566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:21.463754 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:21.463905 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.464426 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.464556 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:21:21.465549 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.453082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34269 (local address 127.0.98.2:46633)
0620 14:21:21.453599 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:21.453647 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:21.454843 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:21.455362 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:21.455369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:21.455866 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:21.456371 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.456385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.457135 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.457138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.458025 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.458033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.458560 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.458563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.458703 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.459247 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:21.459263 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:21.460976 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:21.462725 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:21.462731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:21.462746 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:21.463018 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:21.463313 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:21.463317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:21.463319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:21.463463 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:21.463913 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:21.463922 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:21.464186 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 14:21:21.464378 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 14:21:21.464709 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":127,"threads_started":1}
I20260620 14:21:21.466816 10957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34269
I20260620 14:21:21.467067 10957 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:21.467548 10957 heartbeater.cc:511] Master 127.0.98.62:34269 requested a full tablet report, sending...
I20260620 14:21:21.468699 10632 ts_manager.cc:194] Registered new tserver with Master: b97c2b61e3d940198d08e7421064d348 (127.0.98.2:38305)
I20260620 14:21:21.469271 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46633
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:21:21.488555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:21.599120 10962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:21.599409 10962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:21.599454 10962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:21.603509 10962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:21.603574 10962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:21.603693 10962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:21.609341 10962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:21:21.610407 10962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:21.611474 10962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:21.621106 10968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:21.621147 10970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:21.621163 10967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:21.621521 10962 server_base.cc:1034] running on GCE node
I20260620 14:21:21.621924 10962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:21.622550 10962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:21.623705 10962 hybrid_clock.cc:648] HybridClock initialized: now 1781965281623692 us; error 27 us; skew 500 ppm
Jun 20 14:21:21 dist-test-slave-87nc krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:21.626485 10962 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:21:21.627346 10962 webserver.cc:460] Webserver started at http://127.0.98.3:42267/ using document root <none> and password file <none>
I20260620 14:21:21.627913 10962 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:21.627978 10962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:21.628151 10962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:21.630134 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a"
format_stamp: "Formatted at 2026-06-20 14:21:21 on dist-test-slave-87nc"
server_key: "ff01c3497caede482a7539aeab06df85"
server_key_iv: "610f6306c9e237f349fed12414c4da4f"
server_key_version: "encryptionkey@0"
I20260620 14:21:21.630635 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a"
format_stamp: "Formatted at 2026-06-20 14:21:21 on dist-test-slave-87nc"
server_key: "ff01c3497caede482a7539aeab06df85"
server_key_iv: "610f6306c9e237f349fed12414c4da4f"
server_key_version: "encryptionkey@0"
I20260620 14:21:21.634370 10962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:21:21.636708 10977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:21.637957 10962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:21:21.638054 10962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a"
format_stamp: "Formatted at 2026-06-20 14:21:21 on dist-test-slave-87nc"
server_key: "ff01c3497caede482a7539aeab06df85"
server_key_iv: "610f6306c9e237f349fed12414c4da4f"
server_key_version: "encryptionkey@0"
I20260620 14:21:21.638149 10962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:21.673717 10962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:21.677191 10962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:21.677343 10962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:21.677850 10962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:21.678601 10962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:21.678646 10962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:21.678682 10962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:21.678697 10962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:21.688037 10962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44225
I20260620 14:21:21.688062 11090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44225 every 8 connection(s)
I20260620 14:21:21.688934 10962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:21:21.695338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 10962
I20260620 14:21:21.695474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:21:21.695856 392 external_mini_cluster.cc:1428] Setting key d52be9635684f462005f1384812cf5af
Jun 20 14:21:21 dist-test-slave-87nc krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:21.703552 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.690653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45969 (local address 127.0.98.62:34269)
0620 14:21:21.690782 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.690786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.691700 (+ 914us) server_negotiation.cc:407] Connection header received
0620 14:21:21.693132 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.693135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.693217 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.693330 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:21.694512 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.695022 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.696826 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.697076 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.697188 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.700216 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:21.700219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:21.700221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:21.700244 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:21.701706 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:21.702162 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:21.702165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:21.702166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:21.702240 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:21.702597 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:21.702599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:21.702600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:21.702776 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:21.702848 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.703338 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.703433 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:21:21.704541 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.691113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34269 (local address 127.0.98.3:45969)
0620 14:21:21.691557 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:21.691597 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:21.692877 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:21.693520 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:21.693528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:21.693902 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:21.694375 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.694389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.695163 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.695167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.696612 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.696621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.697263 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.697268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.697522 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.698309 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:21.698326 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:21.700063 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:21.701803 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:21.701807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:21.701819 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:21.702070 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:21.702341 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:21.702344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:21.702345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:21.702503 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:21.702867 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:21.702874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:21.703121 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 14:21:21.703292 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 14:21:21.703595 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":124,"threads_started":1}
I20260620 14:21:21.705711 11091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34269
I20260620 14:21:21.706002 11091 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:21.706525 11091 heartbeater.cc:511] Master 127.0.98.62:34269 requested a full tablet report, sending...
I20260620 14:21:21.707571 10632 ts_manager.cc:194] Registered new tserver with Master: fa4d1c7041f34b7a9c3d8ad5f1bc1c9a (127.0.98.3:44225)
I20260620 14:21:21.708195 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45969
I20260620 14:21:21.709941 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:21:21.719527 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.711840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.0.98.62:34269)
0620 14:21:21.711959 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.711964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.712152 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:21:21.712312 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.712315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.712363 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.712444 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:21.713180 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.713695 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.714570 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.714793 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.714898 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.716356 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:21.716360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:21.716362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:21.716387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:21.717816 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:21.718240 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:21.718244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:21.718246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:21.718327 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:21.718598 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:21.718603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:21.718605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:21.718820 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:21.718960 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.719252 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.719364 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:21:21.722388 10632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:21:21.725435 10632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:21:21.738888 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.733320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44225 (local address 127.0.0.1:40726)
0620 14:21:21.733678 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:21.733727 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:21.733847 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:21.734902 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:21.734908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:21.734931 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:21.735041 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.735046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.736254 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.736258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.737241 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.737249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.738322 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.738326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.738479 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.738526 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:21:21.738644 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:21:21.738763 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"thread_start_us":122,"threads_started":1}
I20260620 14:21:21.739372 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.733801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.0.98.3:44225)
0620 14:21:21.734253 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.734266 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.734292 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:21:21.734373 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.734379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.734634 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.734795 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:21.735158 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.736113 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.737371 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.738197 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.738303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.738463 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.739118 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.739233 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":136,"threads_started":1}
I20260620 14:21:21.740115 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.734104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38305 (local address 127.0.0.1:42300)
0620 14:21:21.734666 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:21.734677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:21.734772 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:21.735540 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:21.735543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:21.735565 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:21.735654 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.735660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.737241 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.737244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.738585 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.738594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.739760 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.739802 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.739886 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.739905 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:21:21.739966 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:21:21.740015 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":121,"thread_start_us":98,"threads_started":1}
I20260620 14:21:21.740219 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.734278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.0.98.2:38305)
0620 14:21:21.735170 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.735179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.735197 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:21:21.735257 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.735261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.735403 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.735514 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:21.735804 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.736981 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.738852 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.739643 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.739753 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.739910 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.739992 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.740075 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":134,"threads_started":1}
I20260620 14:21:21.742283 11025 tablet_service.cc:1467] Processing CreateTablet for tablet b4156f7918a040c0b7ec5e36c27a75a3 (DEFAULT_TABLE table=test-table [id=86a0df91626447819acd1688cac5fadd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:21.742794 10891 tablet_service.cc:1467] Processing CreateTablet for tablet b4156f7918a040c0b7ec5e36c27a75a3 (DEFAULT_TABLE table=test-table [id=86a0df91626447819acd1688cac5fadd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:21.743259 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.733391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42927 (local address 127.0.0.1:51306)
0620 14:21:21.734387 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:21.734401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:21.734484 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:21.736106 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:21.736108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:21.736127 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:21.736246 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.736252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.737563 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.737566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.738538 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.738546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.742857 (+ 4311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.742861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.742957 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.742979 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:21:21.743060 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:21:21.743125 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":23,"thread_start_us":94,"threads_started":1}
I20260620 14:21:21.743463 11025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4156f7918a040c0b7ec5e36c27a75a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:21.743466 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.734059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.0.98.1:42927)
0620 14:21:21.735650 (+ 1591us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.735655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.735673 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:21:21.735761 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.735766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.735965 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.736119 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:21.736360 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.737443 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.738776 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.742694 (+ 3918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.742811 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.742936 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.743104 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.743198 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":78,"threads_started":1}
I20260620 14:21:21.743911 10891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4156f7918a040c0b7ec5e36c27a75a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:21.746158 10757 tablet_service.cc:1467] Processing CreateTablet for tablet b4156f7918a040c0b7ec5e36c27a75a3 (DEFAULT_TABLE table=test-table [id=86a0df91626447819acd1688cac5fadd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:21.747148 10757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4156f7918a040c0b7ec5e36c27a75a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:21.749586 11108 tablet_bootstrap.cc:492] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a: Bootstrap starting.
I20260620 14:21:21.752125 11108 tablet_bootstrap.cc:654] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:21.753248 11108 log.cc:824] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:21.753996 11109 tablet_bootstrap.cc:492] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: Bootstrap starting.
I20260620 14:21:21.754379 11111 tablet_bootstrap.cc:492] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c: Bootstrap starting.
I20260620 14:21:21.755270 11108 tablet_bootstrap.cc:492] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a: No bootstrap required, opened a new log
I20260620 14:21:21.755496 11108 ts_tablet_manager.cc:1397] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:21:21.756556 11111 tablet_bootstrap.cc:654] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:21.756976 11109 tablet_bootstrap.cc:654] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:21.757470 11111 log.cc:824] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:21.757750 11109 log.cc:824] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:21.759640 11109 tablet_bootstrap.cc:492] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: No bootstrap required, opened a new log
I20260620 14:21:21.759640 11111 tablet_bootstrap.cc:492] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c: No bootstrap required, opened a new log
I20260620 14:21:21.759554 11108 raft_consensus.cc:348] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } }
I20260620 14:21:21.759857 11111 ts_tablet_manager.cc:1397] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:21:21.759857 11109 ts_tablet_manager.cc:1397] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:21:21.759873 11108 raft_consensus.cc:374] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:21.759934 11108 raft_consensus.cc:729] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4d1c7041f34b7a9c3d8ad5f1bc1c9a, State: Initialized, Role: FOLLOWER
I20260620 14:21:21.760406 11108 consensus_queue.cc:260] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } }
I20260620 14:21:21.761071 11091 heartbeater.cc:503] Master 127.0.98.62:34269 was elected leader, sending a full tablet report...
I20260620 14:21:21.761437 11108 ts_tablet_manager.cc:1428] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:21:21.764274 11111 raft_consensus.cc:348] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } }
I20260620 14:21:21.764554 11111 raft_consensus.cc:374] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:21.764617 11111 raft_consensus.cc:729] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a346359e1254b888cf1fe0254466e7c, State: Initialized, Role: FOLLOWER
I20260620 14:21:21.764873 11109 raft_consensus.cc:348] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } }
I20260620 14:21:21.765059 11111 consensus_queue.cc:260] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } }
I20260620 14:21:21.765136 11109 raft_consensus.cc:374] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:21.765189 11109 raft_consensus.cc:729] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97c2b61e3d940198d08e7421064d348, State: Initialized, Role: FOLLOWER
I20260620 14:21:21.765588 11109 consensus_queue.cc:260] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } }
I20260620 14:21:21.766062 11111 ts_tablet_manager.cc:1428] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:21:21.766397 10823 heartbeater.cc:503] Master 127.0.98.62:34269 was elected leader, sending a full tablet report...
I20260620 14:21:21.766494 11109 ts_tablet_manager.cc:1428] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:21:21.766786 10957 heartbeater.cc:503] Master 127.0.98.62:34269 was elected leader, sending a full tablet report...
I20260620 14:21:21.845306 11116 raft_consensus.cc:482] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:21.845559 11116 raft_consensus.cc:504] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } }
I20260620 14:21:21.846860 11116 leader_election.cc:288] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa4d1c7041f34b7a9c3d8ad5f1bc1c9a (127.0.98.3:44225), 1a346359e1254b888cf1fe0254466e7c (127.0.98.1:42927)
I20260620 14:21:21.850780 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.847151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42927 (local address 127.0.98.2:33707)
0620 14:21:21.847279 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:21.847296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:21.847404 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:21.847869 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:21.847872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:21.847890 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:21.848004 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.848018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.848748 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.848751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.849592 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.849602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.850360 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.850362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.850442 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.850460 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:21:21.850523 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:21:21.850578 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:21:21.850837 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.847474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33707 (local address 127.0.98.1:42927)
0620 14:21:21.847586 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.847589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.847602 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:21:21.847654 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.847656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.847695 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.847860 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:21.848122 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.848625 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.849715 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.850136 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.850257 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.850310 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.850569 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.850681 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:21:21.851527 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.847425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48983 (local address 127.0.98.3:44225)
0620 14:21:21.847565 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.847571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.847804 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:21:21.847992 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.847995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.848051 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.848135 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:21.848432 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.848960 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.850200 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.850836 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.850951 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.851014 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.851338 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.851408 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:21:21.851547 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.847158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44225 (local address 127.0.98.2:48983)
0620 14:21:21.847674 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:21.847685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:21.847850 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:21.848157 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:21.848160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:21.848177 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:21.848311 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.848317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.849074 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.849081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.850044 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:21.850054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.851095 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.851099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:21.851203 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.851222 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:21:21.851320 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:21:21.851401 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":72,"thread_start_us":128,"threads_started":1}
I20260620 14:21:21.851578 10777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4156f7918a040c0b7ec5e36c27a75a3" candidate_uuid: "b97c2b61e3d940198d08e7421064d348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a346359e1254b888cf1fe0254466e7c" is_pre_election: true
I20260620 14:21:21.851990 10777 raft_consensus.cc:2454] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b97c2b61e3d940198d08e7421064d348 in term 0.
I20260620 14:21:21.852309 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4156f7918a040c0b7ec5e36c27a75a3" candidate_uuid: "b97c2b61e3d940198d08e7421064d348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" is_pre_election: true
I20260620 14:21:21.852497 10846 leader_election.cc:302] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a346359e1254b888cf1fe0254466e7c, b97c2b61e3d940198d08e7421064d348; no voters:
I20260620 14:21:21.852617 11045 raft_consensus.cc:2454] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b97c2b61e3d940198d08e7421064d348 in term 0.
I20260620 14:21:21.852842 11116 raft_consensus.cc:2781] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:21:21.852927 11116 raft_consensus.cc:482] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:21:21.852983 11116 raft_consensus.cc:3037] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:21.854040 11116 raft_consensus.cc:504] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } }
I20260620 14:21:21.854401 11116 leader_election.cc:288] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [CANDIDATE]: Term 1 election: Requested vote from peers fa4d1c7041f34b7a9c3d8ad5f1bc1c9a (127.0.98.3:44225), 1a346359e1254b888cf1fe0254466e7c (127.0.98.1:42927)
I20260620 14:21:21.854851 10777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4156f7918a040c0b7ec5e36c27a75a3" candidate_uuid: "b97c2b61e3d940198d08e7421064d348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a346359e1254b888cf1fe0254466e7c"
I20260620 14:21:21.854839 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4156f7918a040c0b7ec5e36c27a75a3" candidate_uuid: "b97c2b61e3d940198d08e7421064d348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a"
I20260620 14:21:21.854982 11045 raft_consensus.cc:3037] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:21.854982 10777 raft_consensus.cc:3037] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:21.856359 11045 raft_consensus.cc:2454] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b97c2b61e3d940198d08e7421064d348 in term 1.
I20260620 14:21:21.856359 10777 raft_consensus.cc:2454] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b97c2b61e3d940198d08e7421064d348 in term 1.
I20260620 14:21:21.856679 10846 leader_election.cc:302] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a346359e1254b888cf1fe0254466e7c, b97c2b61e3d940198d08e7421064d348; no voters:
I20260620 14:21:21.856926 11116 raft_consensus.cc:2781] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:21.857185 11116 raft_consensus.cc:686] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 1 LEADER]: Becoming Leader. State: Replica: b97c2b61e3d940198d08e7421064d348, State: Running, Role: LEADER
I20260620 14:21:21.857566 11116 consensus_queue.cc:237] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } }
I20260620 14:21:21.860823 10631 catalog_manager.cc:5495] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 reported cstate change: term changed from 0 to 1, leader changed from <none> to b97c2b61e3d940198d08e7421064d348 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b97c2b61e3d940198d08e7421064d348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b97c2b61e3d940198d08e7421064d348" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } health_report { overall_health: HEALTHY } } }
I20260620 14:21:21.895092 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.891505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.0.98.2:38305)
0620 14:21:21.891647 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:21.891650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:21.891684 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:21:21.891755 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:21.891757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:21.891881 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:21.891966 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:21:21.892208 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.892756 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.893603 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:21.893852 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:21.893967 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:21.894270 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:21:21.894699 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:21:21.894834 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:21.894893 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:21:21.894946 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:21:21.900983 11045 raft_consensus.cc:1261] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [term 1 FOLLOWER]: Refusing update from remote peer b97c2b61e3d940198d08e7421064d348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:21.900993 10777 raft_consensus.cc:1261] T b4156f7918a040c0b7ec5e36c27a75a3 P 1a346359e1254b888cf1fe0254466e7c [term 1 FOLLOWER]: Refusing update from remote peer b97c2b61e3d940198d08e7421064d348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:21.901767 11118 consensus_queue.cc:1035] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4d1c7041f34b7a9c3d8ad5f1bc1c9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:21.901969 11116 consensus_queue.cc:1035] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a346359e1254b888cf1fe0254466e7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:21.916798 10631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:21.916985 10631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:21.919185 10631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eced47264d0740b380773bb9a055eb1a: Sending DeleteTablet for 3 replicas of tablet b4156f7918a040c0b7ec5e36c27a75a3
I20260620 14:21:21.919997 10891 tablet_service.cc:1514] Processing DeleteTablet for tablet b4156f7918a040c0b7ec5e36c27a75a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42300
I20260620 14:21:21.919997 11025 tablet_service.cc:1514] Processing DeleteTablet for tablet b4156f7918a040c0b7ec5e36c27a75a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40726
I20260620 14:21:21.920495 10757 tablet_service.cc:1514] Processing DeleteTablet for tablet b4156f7918a040c0b7ec5e36c27a75a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51306
I20260620 14:21:21.920651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 10694
I20260620 14:21:21.921343 11131 tablet_replica.cc:330] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a: stopping tablet replica
I20260620 14:21:21.921648 11131 raft_consensus.cc:2229] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:21:21.921957 11131 raft_consensus.cc:2258] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:21.922360 11132 tablet_replica.cc:330] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: stopping tablet replica
I20260620 14:21:21.922646 11132 raft_consensus.cc:2229] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:21:21.922971 11132 raft_consensus.cc:2258] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:21.924541 11131 ts_tablet_manager.cc:1905] T b4156f7918a040c0b7ec5e36c27a75a3 P fa4d1c7041f34b7a9c3d8ad5f1bc1c9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:21:21.924541 11132 ts_tablet_manager.cc:1905] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:21:21.927230 11132 ts_tablet_manager.cc:1918] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:21.927320 11132 log.cc:1196] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/b4156f7918a040c0b7ec5e36c27a75a3
I20260620 14:21:21.927624 11132 ts_tablet_manager.cc:1939] T b4156f7918a040c0b7ec5e36c27a75a3 P b97c2b61e3d940198d08e7421064d348: Deleting consensus metadata
I20260620 14:21:21.928782 10616 catalog_manager.cc:4841] TS b97c2b61e3d940198d08e7421064d348 (127.0.98.2:38305): tablet b4156f7918a040c0b7ec5e36c27a75a3 (table test-table [id=86a0df91626447819acd1688cac5fadd]) successfully deleted
W20260620 14:21:21.930825 10618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42927 (error 108)
I20260620 14:21:21.931854 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:21.931533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42927 (local address 127.0.0.1:51308)
0620 14:21:21.931673 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:21.931738 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:21:21.931887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 10828
W20260620 14:21:21.931994 10618 catalog_manager.cc:4584] TS 1a346359e1254b888cf1fe0254466e7c (127.0.98.1:42927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4156f7918a040c0b7ec5e36c27a75a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42927: connect: Connection refused (error 111)
I20260620 14:21:21.936961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 10962
I20260620 14:21:21.942354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 10600
2026-06-20T14:21:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2357 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:21:22 dist-test-slave-87nc krb5kdc[11136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:21:22 dist-test-slave-87nc krb5kdc[11136](info): set up 2 sockets
Jun 20 14:21:22 dist-test-slave-87nc krb5kdc[11136](info): commencing operation
krb5kdc: starting...
W20260620 14:21:22.980713 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:21:23 dist-test-slave-87nc krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:21:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:21:23.145890 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUQknKY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:21:25.598 UTC [11166] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:21:25.598 UTC [11166] LOG: listening on IPv4 address "127.0.98.20", port 54533
2026-06-20 14:21:25.601 UTC [11166] LOG: listening on Unix socket "/tmp/.s.PGSQL.54533"
2026-06-20 14:21:25.636 UTC [11169] LOG: database system was shut down at 2026-06-20 14:21:24 UTC
2026-06-20 14:21:25.640 UTC [11166] LOG: database system is ready to accept connections
I20260620 14:21:26.592779 392 mini_postgres.cc:96] Postgres bound to 54533
2026-06-20 14:21:26.599 UTC [11177] FATAL: database "slave" does not exist
127.0.98.20:54533 - accepting connections
I20260620 14:21:26.599598 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:21:26.614133 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:21:27.097046 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:21:27.097144 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-admin
2026-06-20 14:21:27,389 [I] DB FLAVOR :POSTGRES
2026-06-20 14:21:27,390 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:21:27,390 [I] Checking connection..
2026-06-20 14:21:27,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:21:27,641 [I] Checking connection passed.
2026-06-20 14:21:27,641 [I] --------- Verifying version history table ---------
2026-06-20 14:21:27,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:21:27,886 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:21:27,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:21:28,148 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:21:28,148 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:21:28,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:21:28.389 UTC [11241] WARNING: there is no transaction in progress
2026-06-20 14:21:28,395 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:21:28,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:21:28,655 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:21:28,655 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:21:28,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:21:28,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:21:29,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-87nc.c.gcp-upstream.internal','N') ;"
2026-06-20 14:21:29,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:21:29,578 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:21:29,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:21:29,826 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:21:29,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:21:30,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:21:30,330 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:21:30,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:21:31.290 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.307 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.324 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.339 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.358 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.380 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.450 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.457 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.472 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.482 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.493 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.507 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.517 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.524 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.531 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:31.537 UTC [11349] WARNING: there is no transaction in progress
2026-06-20 14:21:32,157 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:21:32,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:21:32,412 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:21:32,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:21:32,673 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:21:32,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:21:32,908 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:21:32,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:21:33,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:21:33,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-87nc.c.gcp-upstream.internal';"
2026-06-20 14:21:33,621 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:21:33,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:21:33,851 [I] DB_PATCHES have already been applied
I20260620 14:21:33.857517 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUQknKY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:21:33.857586 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:21:33.861040 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:41345
Jun 20, 2026 2:21:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:21:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:21:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:21:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:21:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:21:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:21:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:21:35.078909 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.479s user 0.005s sys 0.005s
Jun 20, 2026 2:21:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:21:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 2:21:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:21:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:21:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:21:57.318960 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:21:57 dist-test-slave-87nc krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965317, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:21:57.404843 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:21:57.414542 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:21:58.230361 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:21:58.230469 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-kms
I20260620 14:21:58.230569 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54533
2026-06-20 14:21:58,455 [I] DB FLAVOR :POSTGRES
2026-06-20 14:21:58,455 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:21:58,455 [I] Checking connection
2026-06-20 14:21:58,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:21:58,717 [I] connection success
2026-06-20 14:21:58,718 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:21:58,718 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:21:58,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:21:58,968 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:21:58,968 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:21:58,968 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:21:58,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54533/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:21:59,267 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:21:59.737118 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:21:59.955921 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:21:59.999723 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:21:59 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965317, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:00.400575 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:22:00.400741 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:22:00.400763 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:22:00.404472 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11529
I20260620 14:22:00.404629 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45089
Jun 20, 2026 2:22:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:22:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:22:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:22:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:22:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:22:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:22:02.638188 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.233s user 0.012s sys 0.196s
I20260620 14:22:02.638320 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:22:02.638423 392 mini_ranger_kms.cc:401] 127.0.98.20:45089/kms/v1/keys
Jun 20 14:22:02 dist-test-slave-87nc krb5kdc[11136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965322, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:22:03 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965322, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:22:03 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965322, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:22:05 dist-test-slave-87nc krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:22:05.680773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33605
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33605
--ranger_config_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45089/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:05.806881 11580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:05.807320 11580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:05.807387 11580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:05.815640 11580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:05.815752 11580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:05.815863 11580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:05.815907 11580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:05.815979 11580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:05.827450 11580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45089/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33605
--ranger_config_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:05.828660 11580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:05.829717 11580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:22:05.839838 11580 server_base.cc:1034] running on GCE node
W20260620 14:22:05.839833 11585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:05.839800 11588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:05.840334 11586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:05.841045 11580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:05.842610 11580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:05.843835 11580 hybrid_clock.cc:648] HybridClock initialized: now 1781965325843798 us; error 53 us; skew 500 ppm
Jun 20 14:22:05 dist-test-slave-87nc krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:05.847262 11580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:22:05.848330 11580 webserver.cc:460] Webserver started at http://127.0.98.62:37095/ using document root <none> and password file <none>
I20260620 14:22:05.848923 11580 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:05.849009 11580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:05.849246 11580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:22:05 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:05.895392 11580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "634f295d7de9492b893460629976221a"
format_stamp: "Formatted at 2026-06-20 14:22:05 on dist-test-slave-87nc"
server_key: "fff9770ac49e2718f5b1dd6a02821501"
server_key_iv: "519ea2cf653f0dc6d5ea17e1854bda4f"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:05.896060 11580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "634f295d7de9492b893460629976221a"
format_stamp: "Formatted at 2026-06-20 14:22:05 on dist-test-slave-87nc"
server_key: "fff9770ac49e2718f5b1dd6a02821501"
server_key_iv: "519ea2cf653f0dc6d5ea17e1854bda4f"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:05.900318 11580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:22:05.914651 11595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:05.916231 11580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:22:05.916399 11580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "634f295d7de9492b893460629976221a"
format_stamp: "Formatted at 2026-06-20 14:22:05 on dist-test-slave-87nc"
server_key: "fff9770ac49e2718f5b1dd6a02821501"
server_key_iv: "519ea2cf653f0dc6d5ea17e1854bda4f"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:05.916553 11580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:05.933696 11580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:05.936854 11580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:05.937077 11580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:05.945386 11580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33605
I20260620 14:22:05.945442 11647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33605 every 8 connection(s)
I20260620 14:22:05.946530 11580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:05.948247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 11580
I20260620 14:22:05.948378 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:22:05.948658 392 external_mini_cluster.cc:1428] Setting key d5d35d20eeb40d32df9bf74028a83f2b
I20260620 14:22:05.951140 11648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:22:05 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:05.958226 11648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a: Bootstrap starting.
I20260620 14:22:05.960916 11648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:05.962292 11648 log.cc:824] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:05.963459 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:05.950103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.0.98.62:33605)
0620 14:22:05.950545 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:05.950560 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:05.950595 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:22:05.951318 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:05.951340 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:05.951680 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:05.952090 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:05.952869 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:05.953877 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:05.954777 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:05.955270 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:05.955447 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:05.957998 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:05.958003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:05.958015 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:05.958042 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:05.960176 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:05.960653 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:05.960658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:05.960663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:05.960777 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:05.961106 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:05.961110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:05.961111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:05.961513 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:05.961719 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:05.962032 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:22:05.962496 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":150,"threads_started":1}
I20260620 14:22:05.964679 11648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a: No bootstrap required, opened a new log
I20260620 14:22:05.968627 11648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634f295d7de9492b893460629976221a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } }
I20260620 14:22:05.968900 11648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:05.968946 11648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634f295d7de9492b893460629976221a, State: Initialized, Role: FOLLOWER
I20260620 14:22:05.969424 11648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634f295d7de9492b893460629976221a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } }
I20260620 14:22:05.969584 11648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:22:05.969645 11648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:22:05.969735 11648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:05.970863 11648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634f295d7de9492b893460629976221a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } }
I20260620 14:22:05.971222 11648 leader_election.cc:302] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 634f295d7de9492b893460629976221a; no voters:
I20260620 14:22:05.971596 11648 leader_election.cc:288] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:22:05.971702 11653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:05.971966 11653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [term 1 LEADER]: Becoming Leader. State: Replica: 634f295d7de9492b893460629976221a, State: Running, Role: LEADER
I20260620 14:22:05.972330 11653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634f295d7de9492b893460629976221a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } }
I20260620 14:22:05.972887 11648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:05.973448 11653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 634f295d7de9492b893460629976221a. Latest consensus state: current_term: 1 leader_uuid: "634f295d7de9492b893460629976221a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634f295d7de9492b893460629976221a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } } }
I20260620 14:22:05.973656 11653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:05.973865 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "634f295d7de9492b893460629976221a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634f295d7de9492b893460629976221a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } } }
I20260620 14:22:05.974005 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:05.974103 11662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:05.977281 11662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:22:05.977708 11648 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:22:05.983270 11662 catalog_manager.cc:1347] Generated new cluster ID: ce228a7e9a424cb49a1a9fc5d541880a
I20260620 14:22:05.983341 11662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:22:06.009133 11662 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:22:06.010620 11662 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:06.017267 11662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a: Generated new TSK 0
I20260620 14:22:06.018195 11662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:22:06.802690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:06.960356 11693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:06.960675 11693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:06.960773 11693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:06.965696 11693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:06.965778 11693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:06.965879 11693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:22:06.971206 11693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:06.972345 11693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:06.973618 11693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:06.983708 11700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:06.983969 11703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:06.983748 11701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:06.983855 11693 server_base.cc:1034] running on GCE node
I20260620 14:22:06.984587 11693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:06.985260 11693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:06.986451 11693 hybrid_clock.cc:648] HybridClock initialized: now 1781965326986410 us; error 56 us; skew 500 ppm
Jun 20 14:22:06 dist-test-slave-87nc krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:06.990242 11693 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:22:06.991405 11693 webserver.cc:460] Webserver started at http://127.0.98.1:33863/ using document root <none> and password file <none>
I20260620 14:22:06.992027 11693 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:06.992123 11693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:06.992342 11693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:22:06 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:07.007668 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "f00231cef0c24c26a9d99d137d8f3bef"
format_stamp: "Formatted at 2026-06-20 14:22:07 on dist-test-slave-87nc"
server_key: "0ae74f31d4698e3c64dbbb253908ea2d"
server_key_iv: "5d2535b52c926b3740a9feb78badb7a3"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:07.008538 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f00231cef0c24c26a9d99d137d8f3bef"
format_stamp: "Formatted at 2026-06-20 14:22:07 on dist-test-slave-87nc"
server_key: "0ae74f31d4698e3c64dbbb253908ea2d"
server_key_iv: "5d2535b52c926b3740a9feb78badb7a3"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:07.013562 11693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:22:07.045825 11710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:07.047717 11693 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.000s sys 0.003s
I20260620 14:22:07.047892 11693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "f00231cef0c24c26a9d99d137d8f3bef"
format_stamp: "Formatted at 2026-06-20 14:22:07 on dist-test-slave-87nc"
server_key: "0ae74f31d4698e3c64dbbb253908ea2d"
server_key_iv: "5d2535b52c926b3740a9feb78badb7a3"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:07.048043 11693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:07.062000 11693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:07.065750 11693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:07.065995 11693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:07.066694 11693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:07.067795 11693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:07.067907 11693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:07.068082 11693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:07.068166 11693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:07.080070 11693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34877
I20260620 14:22:07.081300 11693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:22:07.081833 11825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34877 every 8 connection(s)
I20260620 14:22:07.090044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 11693
I20260620 14:22:07.090167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:22:07.090456 392 external_mini_cluster.cc:1428] Setting key 20cd651bfe43a4164ef1910f1322c007
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 14:22:07 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:07.120052 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.083995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48841 (local address 127.0.98.62:33605)
0620 14:22:07.084246 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:07.084251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:07.102257 (+ 18006us) server_negotiation.cc:407] Connection header received
0620 14:22:07.104699 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:07.104702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:07.104780 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:07.104871 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:07.106501 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.107385 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.109811 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.110224 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.110317 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.115529 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:07.115533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:07.115536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:07.115566 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:07.117440 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:07.118108 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:07.118113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:07.118115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:07.118205 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:07.118647 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:07.118650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:07.118652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:07.118891 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:07.118983 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:07.119687 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 14:22:07.119854 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":57,"threads_started":1}
I20260620 14:22:07.122035 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.084519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33605 (local address 127.0.98.1:48841)
0620 14:22:07.101051 (+ 16532us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:07.101140 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:07.104281 (+ 3141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:07.105229 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:07.105241 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:07.105603 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:07.106322 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.106342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.108073 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.108078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.109461 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.109472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.110543 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.110547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.110871 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.113471 (+ 2600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:07.113493 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:07.115307 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:07.117594 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:07.117600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:07.117618 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:07.117972 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:07.118344 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:07.118347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:07.118349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:07.118531 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:07.119050 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:07.119056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:07.119408 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 14:22:07.120531 (+ 1123us) client_negotiation.cc:240] Negotiation successful
0620 14:22:07.120905 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16226,"thread_start_us":133,"threads_started":1}
I20260620 14:22:07.123732 11826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33605
I20260620 14:22:07.123996 11826 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:07.124550 11826 heartbeater.cc:511] Master 127.0.98.62:33605 requested a full tablet report, sending...
I20260620 14:22:07.126295 11612 ts_manager.cc:194] Registered new tserver with Master: f00231cef0c24c26a9d99d137d8f3bef (127.0.98.1:34877)
I20260620 14:22:07.127808 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:22:07.148130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:22:07 dist-test-slave-87nc krb5kdc[11136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:22:07.292908 11832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:07.293342 11832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:07.293506 11832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:07.299664 11832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:07.299757 11832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:07.299892 11832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:22:07.308106 11832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:07.309557 11832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:07.310945 11832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:07.323985 11839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:07.323999 11841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:07.324252 11838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:07.324927 11832 server_base.cc:1034] running on GCE node
I20260620 14:22:07.325371 11832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:07.326103 11832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:07.327328 11832 hybrid_clock.cc:648] HybridClock initialized: now 1781965327327253 us; error 93 us; skew 500 ppm
Jun 20 14:22:07 dist-test-slave-87nc krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:07.330943 11832 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:22:07.332103 11832 webserver.cc:460] Webserver started at http://127.0.98.2:34967/ using document root <none> and password file <none>
I20260620 14:22:07.332800 11832 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:07.332890 11832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:07.333107 11832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:22:07 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:07.348146 11832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "a5d17567ac6e44099080cff86f9001bf"
format_stamp: "Formatted at 2026-06-20 14:22:07 on dist-test-slave-87nc"
server_key: "5859bdef4be81cfa6130ed463bdf9d07"
server_key_iv: "31b1635f660b3344fa0af1378e38c29c"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:07.349146 11832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a5d17567ac6e44099080cff86f9001bf"
format_stamp: "Formatted at 2026-06-20 14:22:07 on dist-test-slave-87nc"
server_key: "5859bdef4be81cfa6130ed463bdf9d07"
server_key_iv: "31b1635f660b3344fa0af1378e38c29c"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:07.361402 11832 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260620 14:22:07.381142 11848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:07.385161 11832 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 14:22:07.385321 11832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "a5d17567ac6e44099080cff86f9001bf"
format_stamp: "Formatted at 2026-06-20 14:22:07 on dist-test-slave-87nc"
server_key: "5859bdef4be81cfa6130ed463bdf9d07"
server_key_iv: "31b1635f660b3344fa0af1378e38c29c"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:07.385495 11832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:07.404843 11832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:07.408740 11832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:07.408947 11832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:07.409667 11832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:07.410812 11832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:07.410879 11832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:07.410941 11832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:07.410974 11832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:07.421977 11832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39103
I20260620 14:22:07.423180 11832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:22:07.428968 11961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39103 every 8 connection(s)
I20260620 14:22:07.431533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 11832
I20260620 14:22:07.431672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:22:07.432003 392 external_mini_cluster.cc:1428] Setting key 727397c561c236d04b1ac76c11f5b72d
Jun 20 14:22:07 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:07.450560 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.432468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41763 (local address 127.0.98.62:33605)
0620 14:22:07.432658 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:07.432663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:07.433229 (+ 566us) server_negotiation.cc:407] Connection header received
0620 14:22:07.434489 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:07.434493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:07.434549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:07.434609 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:07.436383 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.437227 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.442166 (+ 4939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.442511 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.442596 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.445805 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:07.445809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:07.445812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:07.445843 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:07.447628 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:07.448404 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:07.448408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:07.448410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:07.448521 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:07.449053 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:07.449057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:07.449059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:07.449313 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:07.449409 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:07.450214 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 14:22:07.450374 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 14:22:07.452383 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.430146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33605 (local address 127.0.98.2:41763)
0620 14:22:07.433037 (+ 2891us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:07.433111 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:07.434226 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:07.435034 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:07.435043 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:07.435426 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:07.436208 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.436228 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.437383 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.437387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.441994 (+ 4607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.442008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.442818 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.442821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.443036 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.443819 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:07.443841 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:07.445598 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:07.447807 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:07.447813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:07.447835 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:07.448242 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:07.448652 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:07.448655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:07.448657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:07.448865 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:07.449486 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:07.449494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:07.449889 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 14:22:07.450781 (+ 892us) client_negotiation.cc:240] Negotiation successful
0620 14:22:07.451192 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2662,"thread_start_us":174,"threads_started":1}
I20260620 14:22:07.453891 11962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33605
I20260620 14:22:07.454152 11962 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:07.454751 11962 heartbeater.cc:511] Master 127.0.98.62:33605 requested a full tablet report, sending...
I20260620 14:22:07.455973 11612 ts_manager.cc:194] Registered new tserver with Master: a5d17567ac6e44099080cff86f9001bf (127.0.98.2:39103)
I20260620 14:22:07.456719 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41763
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:22:07.486104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:07.654297 11967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:07.654661 11967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:07.654731 11967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:07.660807 11967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:07.660897 11967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:07.661006 11967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:22:07.669108 11967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:07.670519 11967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:07.671922 11967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:07.685473 11973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:07.685606 11972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:07.685912 11967 server_base.cc:1034] running on GCE node
W20260620 14:22:07.686276 11975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:07.686739 11967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:07.687490 11967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:07.688683 11967 hybrid_clock.cc:648] HybridClock initialized: now 1781965327688618 us; error 100 us; skew 500 ppm
Jun 20 14:22:07 dist-test-slave-87nc krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:07.692889 11967 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:22:07.693887 11967 webserver.cc:460] Webserver started at http://127.0.98.3:38169/ using document root <none> and password file <none>
I20260620 14:22:07.694561 11967 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:07.694641 11967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:07.694864 11967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:22:07 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:07.713771 11967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "94cda8c1daee4da381793ce8b63cfc96"
format_stamp: "Formatted at 2026-06-20 14:22:07 on dist-test-slave-87nc"
server_key: "ee177d4f7c76cc761b30a7e827751bbb"
server_key_iv: "38384adccd5baf4c34d65370426aa300"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:07.714452 11967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "94cda8c1daee4da381793ce8b63cfc96"
format_stamp: "Formatted at 2026-06-20 14:22:07 on dist-test-slave-87nc"
server_key: "ee177d4f7c76cc761b30a7e827751bbb"
server_key_iv: "38384adccd5baf4c34d65370426aa300"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:07.720347 11967 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
Jun 20 14:22:07 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:07.750947 11982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:07.753306 11967 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260620 14:22:07.753451 11967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "94cda8c1daee4da381793ce8b63cfc96"
format_stamp: "Formatted at 2026-06-20 14:22:07 on dist-test-slave-87nc"
server_key: "ee177d4f7c76cc761b30a7e827751bbb"
server_key_iv: "38384adccd5baf4c34d65370426aa300"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:07.753643 11967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:07.790974 11967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:07.809161 11967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:07.809440 11967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:07.810160 11967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:07.811242 11967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:07.811337 11967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:07.811421 11967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:07.811472 11967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:07.821247 11967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33621
I20260620 14:22:07.832054 11967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:22:07.833213 12095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33621 every 8 connection(s)
I20260620 14:22:07.835395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 11967
I20260620 14:22:07.835526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:22:07.835835 392 external_mini_cluster.cc:1428] Setting key c43d5765565ce65c311a8dc20d5f3191
Jun 20 14:22:07 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:07.853385 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.836851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60037 (local address 127.0.98.62:33605)
0620 14:22:07.838668 (+ 1817us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:07.838673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:07.839505 (+ 832us) server_negotiation.cc:407] Connection header received
0620 14:22:07.840769 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:07.840772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:07.840826 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:07.840875 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:07.842710 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.843527 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.845196 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.845534 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.845624 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.848697 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:07.848701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:07.848704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:07.848730 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:07.850474 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:07.851397 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:07.851401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:07.851402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:07.851489 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:07.851916 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:07.851922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:07.851924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:07.852162 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:07.852244 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:07.853097 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 14:22:07.853215 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1725}
I20260620 14:22:07.855223 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.837374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33605 (local address 127.0.98.3:60037)
0620 14:22:07.839353 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:07.839412 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:07.840497 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:07.841416 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:07.841425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:07.841782 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:07.842552 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.842571 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.843677 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.843681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.845064 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.845074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.845834 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.845838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.846046 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.846759 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:07.846780 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:07.848504 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:07.850629 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:07.850635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:07.850653 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:07.851015 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:07.851599 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:07.851602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:07.851604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:07.851805 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:07.852484 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:07.852490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:07.852811 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 14:22:07.853841 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0620 14:22:07.854195 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"thread_start_us":151,"threads_started":1}
I20260620 14:22:07.856578 12096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33605
I20260620 14:22:07.856799 12096 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:07.857381 12096 heartbeater.cc:511] Master 127.0.98.62:33605 requested a full tablet report, sending...
I20260620 14:22:07.858516 11612 ts_manager.cc:194] Registered new tserver with Master: 94cda8c1daee4da381793ce8b63cfc96 (127.0.98.3:33621)
I20260620 14:22:07.859288 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60037
I20260620 14:22:07.865222 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:22:07.878288 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.867960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.0.98.62:33605)
0620 14:22:07.868095 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:07.868100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:07.868444 (+ 344us) server_negotiation.cc:407] Connection header received
0620 14:22:07.868623 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:07.868627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:07.868672 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:07.868754 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:07.869616 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.870411 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.871658 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.872023 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.872120 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.874043 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:07.874047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:07.874051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:07.874076 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:07.876295 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:07.876825 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:07.876828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:07.876830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:07.876919 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:07.877221 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:07.877224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:07.877225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:07.877430 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:07.877556 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:07.878011 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 14:22:07.878129 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:22:07.881951 11612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:22:07.887032 11612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:22:07.906358 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.898835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33621 (local address 127.0.0.1:60218)
0620 14:22:07.900194 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:07.900206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:07.900284 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:07.901222 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:07.901228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:07.901249 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:07.901363 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.901369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.902926 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.902929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.904523 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.904534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.905808 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.905812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.905914 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.905950 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:22:07.906067 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:22:07.906194 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":33,"thread_start_us":71,"threads_started":1}
I20260620 14:22:07.906905 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.897830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39103 (local address 127.0.0.1:40064)
0620 14:22:07.899377 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:07.899427 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:07.899630 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:07.901556 (+ 1926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:07.901559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:07.901575 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:07.901673 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.901678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.903093 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.903096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.905143 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.905154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.906511 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.906515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.906615 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.906636 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:22:07.906703 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:22:07.906769 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"thread_start_us":95,"threads_started":1}
I20260620 14:22:07.908175 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.898037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.0.98.2:39103)
0620 14:22:07.899083 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:07.899101 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:07.899521 (+ 420us) server_negotiation.cc:407] Connection header received
0620 14:22:07.901055 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:07.901061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:07.901266 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:07.901385 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:07.901798 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.902933 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.905314 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.906170 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.906254 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.906406 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:07.907107 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 14:22:07.907215 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":92,"threads_started":1}
I20260620 14:22:07.908766 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.899193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.0.98.3:33621)
0620 14:22:07.900522 (+ 1329us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:07.900536 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:07.900560 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:22:07.900684 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:07.900693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:07.900910 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:07.901068 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:07.901479 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.902788 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.904676 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.905481 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.905588 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.905747 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:07.908511 (+ 2764us) server_negotiation.cc:299] Negotiation successful
0620 14:22:07.908616 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":127,"threads_started":1}
I20260620 14:22:07.911834 11896 tablet_service.cc:1467] Processing CreateTablet for tablet 5c73b4c90c584e3d80ffe589a31ae38f (DEFAULT_TABLE table=test-table [id=8e693f343fab43dfa1e35aec62107266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:07.911837 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 5c73b4c90c584e3d80ffe589a31ae38f (DEFAULT_TABLE table=test-table [id=8e693f343fab43dfa1e35aec62107266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:07.913000 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c73b4c90c584e3d80ffe589a31ae38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:07.913002 11896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c73b4c90c584e3d80ffe589a31ae38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:07.924109 12115 tablet_bootstrap.cc:492] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf: Bootstrap starting.
I20260620 14:22:07.926610 12115 tablet_bootstrap.cc:654] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:07.926627 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.899892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34877 (local address 127.0.0.1:44262)
0620 14:22:07.906929 (+ 7037us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:07.906942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:07.907267 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:07.908147 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:07.908150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:07.908165 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:07.908377 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.908384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.923745 (+ 15361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.923749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.924971 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.924983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.926225 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.926229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.926316 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.926339 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:22:07.926407 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:22:07.926484 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6985,"thread_start_us":85,"threads_started":1}
I20260620 14:22:07.927443 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.900261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.0.98.1:34877)
0620 14:22:07.907527 (+ 7266us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:07.907535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:07.907558 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:22:07.907637 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:07.907643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:07.907870 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:07.908016 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:07.920325 (+ 12309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.921582 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.925138 (+ 3556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.925919 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.926026 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:07.926166 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:07.927193 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0620 14:22:07.927299 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7142,"thread_start_us":85,"threads_started":1}
I20260620 14:22:07.928215 12115 log.cc:824] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:07.930559 11758 tablet_service.cc:1467] Processing CreateTablet for tablet 5c73b4c90c584e3d80ffe589a31ae38f (DEFAULT_TABLE table=test-table [id=8e693f343fab43dfa1e35aec62107266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:07.931550 12115 tablet_bootstrap.cc:492] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf: No bootstrap required, opened a new log
I20260620 14:22:07.931689 11758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c73b4c90c584e3d80ffe589a31ae38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:07.931742 12115 ts_tablet_manager.cc:1397] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 14:22:07.935915 12115 raft_consensus.cc:348] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 14:22:07.936198 12115 raft_consensus.cc:374] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:07.936255 12115 raft_consensus.cc:729] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d17567ac6e44099080cff86f9001bf, State: Initialized, Role: FOLLOWER
I20260620 14:22:07.936780 12115 consensus_queue.cc:260] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 14:22:07.938817 12115 ts_tablet_manager.cc:1428] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:22:07.939190 11962 heartbeater.cc:503] Master 127.0.98.62:33605 was elected leader, sending a full tablet report...
I20260620 14:22:07.940953 12114 tablet_bootstrap.cc:492] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: Bootstrap starting.
I20260620 14:22:07.944305 12114 tablet_bootstrap.cc:654] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:07.944747 12118 tablet_bootstrap.cc:492] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef: Bootstrap starting.
I20260620 14:22:07.947039 12118 tablet_bootstrap.cc:654] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:07.948977 12114 log.cc:824] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:07.950405 12118 log.cc:824] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:07.952960 12114 tablet_bootstrap.cc:492] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: No bootstrap required, opened a new log
I20260620 14:22:07.953255 12114 ts_tablet_manager.cc:1397] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20260620 14:22:07.957571 12114 raft_consensus.cc:348] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 14:22:07.957937 12114 raft_consensus.cc:374] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:07.958050 12114 raft_consensus.cc:729] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94cda8c1daee4da381793ce8b63cfc96, State: Initialized, Role: FOLLOWER
I20260620 14:22:07.958568 12114 consensus_queue.cc:260] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 14:22:07.959386 12096 heartbeater.cc:503] Master 127.0.98.62:33605 was elected leader, sending a full tablet report...
I20260620 14:22:07.959712 12114 ts_tablet_manager.cc:1428] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 14:22:07.962863 12118 tablet_bootstrap.cc:492] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef: No bootstrap required, opened a new log
I20260620 14:22:07.963094 12118 ts_tablet_manager.cc:1397] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260620 14:22:07.968367 12118 raft_consensus.cc:348] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 14:22:07.968684 12118 raft_consensus.cc:374] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:07.968791 12118 raft_consensus.cc:729] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00231cef0c24c26a9d99d137d8f3bef, State: Initialized, Role: FOLLOWER
I20260620 14:22:07.969302 12118 consensus_queue.cc:260] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 14:22:07.970093 11826 heartbeater.cc:503] Master 127.0.98.62:33605 was elected leader, sending a full tablet report...
I20260620 14:22:07.971546 12118 ts_tablet_manager.cc:1428] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 14:22:07.993383 12121 raft_consensus.cc:482] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:07.993633 12121 raft_consensus.cc:504] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 14:22:07.995141 12121 leader_election.cc:288] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5d17567ac6e44099080cff86f9001bf (127.0.98.2:39103), f00231cef0c24c26a9d99d137d8f3bef (127.0.98.1:34877)
I20260620 14:22:08.001354 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.995490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34877 (local address 127.0.98.3:45199)
0620 14:22:07.995625 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:07.995643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:07.995757 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:07.996115 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:07.996119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:07.996140 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:07.996269 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.996276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.997340 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.997344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:07.999825 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.999836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:08.000877 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:08.000881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:08.001025 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:08.001049 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:22:08.001125 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:22:08.001208 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:22:08.002103 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.995704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45199 (local address 127.0.98.1:34877)
0620 14:22:07.995852 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:07.995856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:07.995873 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:22:07.995925 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:07.995927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:07.995978 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:07.996069 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:07.996412 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.997204 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:08.000001 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:08.000638 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:08.000748 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:08.000816 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:08.001885 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0620 14:22:08.001972 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:22:08.002763 11780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c73b4c90c584e3d80ffe589a31ae38f" candidate_uuid: "94cda8c1daee4da381793ce8b63cfc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00231cef0c24c26a9d99d137d8f3bef" is_pre_election: true
I20260620 14:22:08.003120 11780 raft_consensus.cc:2454] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94cda8c1daee4da381793ce8b63cfc96 in term 0.
I20260620 14:22:08.003805 11985 leader_election.cc:302] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94cda8c1daee4da381793ce8b63cfc96, f00231cef0c24c26a9d99d137d8f3bef; no voters:
I20260620 14:22:08.004665 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.996667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36143 (local address 127.0.98.2:39103)
0620 14:22:07.996771 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:07.996775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:07.998125 (+ 1350us) server_negotiation.cc:407] Connection header received
0620 14:22:07.998410 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:07.998413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:07.998493 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:07.998569 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:07.999125 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.999844 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:08.001419 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:08.002179 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:08.002284 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:08.002344 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:08.004473 (+ 2129us) server_negotiation.cc:299] Negotiation successful
0620 14:22:08.004557 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:22:08.004631 12121 raft_consensus.cc:2781] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:22:08.004737 12121 raft_consensus.cc:482] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:22:08.004788 12121 raft_consensus.cc:3037] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:08.005671 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:07.996318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39103 (local address 127.0.98.3:36143)
0620 14:22:07.997981 (+ 1663us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:07.998029 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:07.998218 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:07.998756 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:07.998759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:07.998774 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:07.998943 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:07.998973 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:07.999985 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:07.999998 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:08.001277 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:08.001288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:08.002432 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:08.004219 (+ 1787us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:08.004328 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:08.004354 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:22:08.005010 (+ 656us) client_negotiation.cc:240] Negotiation successful
0620 14:22:08.005091 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1561,"thread_start_us":105,"threads_started":1}
I20260620 14:22:08.006420 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c73b4c90c584e3d80ffe589a31ae38f" candidate_uuid: "94cda8c1daee4da381793ce8b63cfc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d17567ac6e44099080cff86f9001bf" is_pre_election: true
I20260620 14:22:08.006738 11916 raft_consensus.cc:2454] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94cda8c1daee4da381793ce8b63cfc96 in term 0.
I20260620 14:22:08.007556 12121 raft_consensus.cc:504] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 14:22:08.008322 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c73b4c90c584e3d80ffe589a31ae38f" candidate_uuid: "94cda8c1daee4da381793ce8b63cfc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d17567ac6e44099080cff86f9001bf"
I20260620 14:22:08.008445 11916 raft_consensus.cc:3037] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:08.009047 11780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c73b4c90c584e3d80ffe589a31ae38f" candidate_uuid: "94cda8c1daee4da381793ce8b63cfc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00231cef0c24c26a9d99d137d8f3bef"
I20260620 14:22:08.009173 11780 raft_consensus.cc:3037] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:08.009850 11916 raft_consensus.cc:2454] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94cda8c1daee4da381793ce8b63cfc96 in term 1.
I20260620 14:22:08.010191 11986 leader_election.cc:302] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94cda8c1daee4da381793ce8b63cfc96, a5d17567ac6e44099080cff86f9001bf; no voters:
I20260620 14:22:08.010411 11780 raft_consensus.cc:2454] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94cda8c1daee4da381793ce8b63cfc96 in term 1.
I20260620 14:22:08.010556 12124 raft_consensus.cc:2781] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:08.010776 12124 raft_consensus.cc:686] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 1 LEADER]: Becoming Leader. State: Replica: 94cda8c1daee4da381793ce8b63cfc96, State: Running, Role: LEADER
I20260620 14:22:08.010854 12121 leader_election.cc:288] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [CANDIDATE]: Term 1 election: Requested vote from peers a5d17567ac6e44099080cff86f9001bf (127.0.98.2:39103), f00231cef0c24c26a9d99d137d8f3bef (127.0.98.1:34877)
I20260620 14:22:08.011237 12124 consensus_queue.cc:237] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 14:22:08.015033 11612 catalog_manager.cc:5495] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94cda8c1daee4da381793ce8b63cfc96 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "94cda8c1daee4da381793ce8b63cfc96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94cda8c1daee4da381793ce8b63cfc96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } health_report { overall_health: UNKNOWN } } }
I20260620 14:22:08.034951 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:08.029737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.0.98.3:33621)
0620 14:22:08.029878 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:08.029882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:08.029893 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:22:08.029965 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:08.029967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:08.030021 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:08.030156 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:22:08.030440 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:08.031252 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:08.032573 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:08.032919 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:08.033027 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:08.033457 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:22:08.034099 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:22:08.034231 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:08.034648 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 14:22:08.034741 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:22:08.042459 11916 raft_consensus.cc:1261] T 5c73b4c90c584e3d80ffe589a31ae38f P a5d17567ac6e44099080cff86f9001bf [term 1 FOLLOWER]: Refusing update from remote peer 94cda8c1daee4da381793ce8b63cfc96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:08.042459 11780 raft_consensus.cc:1261] T 5c73b4c90c584e3d80ffe589a31ae38f P f00231cef0c24c26a9d99d137d8f3bef [term 1 FOLLOWER]: Refusing update from remote peer 94cda8c1daee4da381793ce8b63cfc96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:08.043346 12124 consensus_queue.cc:1035] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d17567ac6e44099080cff86f9001bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:08.044126 12124 consensus_queue.cc:1035] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00231cef0c24c26a9d99d137d8f3bef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:08.052582 12129 mvcc.cc:205] Tried to move back new op lower bound from 7298929983652945920 to 7298929983537569792. Current Snapshot: MvccSnapshot[applied={T|T < 7298929983652945920}]
Jun 20 14:22:08 dist-test-slave-87nc krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:08.069871 11612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:08.070039 11612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:08.071412 12132 mvcc.cc:205] Tried to move back new op lower bound from 7298929983652945920 to 7298929983537569792. Current Snapshot: MvccSnapshot[applied={T|T < 7298929983652945920}]
I20260620 14:22:08.072973 11612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 634f295d7de9492b893460629976221a: Sending DeleteTablet for 3 replicas of tablet 5c73b4c90c584e3d80ffe589a31ae38f
I20260620 14:22:08.074535 11896 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c73b4c90c584e3d80ffe589a31ae38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40064
I20260620 14:22:08.075050 11758 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c73b4c90c584e3d80ffe589a31ae38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44262
I20260620 14:22:08.075131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 11693
I20260620 14:22:08.076033 12030 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c73b4c90c584e3d80ffe589a31ae38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60218
I20260620 14:22:08.076485 12138 tablet_replica.cc:330] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: stopping tablet replica
I20260620 14:22:08.076831 12138 raft_consensus.cc:2229] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:22:08.077234 12138 raft_consensus.cc:2258] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:08.079316 12138 ts_tablet_manager.cc:1905] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:08.084229 12138 ts_tablet_manager.cc:1918] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:08.084520 12138 log.cc:1196] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/5c73b4c90c584e3d80ffe589a31ae38f
I20260620 14:22:08.085100 12138 ts_tablet_manager.cc:1939] T 5c73b4c90c584e3d80ffe589a31ae38f P 94cda8c1daee4da381793ce8b63cfc96: Deleting consensus metadata
I20260620 14:22:08.086691 11599 catalog_manager.cc:4841] TS 94cda8c1daee4da381793ce8b63cfc96 (127.0.98.3:33621): tablet 5c73b4c90c584e3d80ffe589a31ae38f (table test-table [id=8e693f343fab43dfa1e35aec62107266]) successfully deleted
W20260620 14:22:08.087656 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34877 (error 108)
I20260620 14:22:08.088577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 11832
I20260620 14:22:08.090396 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:08.090072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34877 (local address 127.0.0.1:44264)
0620 14:22:08.090240 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:08.090309 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 14:22:08.090593 11598 catalog_manager.cc:4584] TS f00231cef0c24c26a9d99d137d8f3bef (127.0.98.1:34877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c73b4c90c584e3d80ffe589a31ae38f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34877: connect: Connection refused (error 111)
W20260620 14:22:08.098659 12112 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:22:08.097986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39103 (local address 127.0.0.1:40080)
0620 14:22:08.098196 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:08.098223 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:08.098318 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:08.098542 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39103: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 14:22:08.099390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 11967
I20260620 14:22:08.109877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 11580
I20260620 14:22:08.119925 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:22:08.465042 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:22:08.465201 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:22:08.810500 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:22:08.810 UTC [11166] LOG: received smart shutdown request
2026-06-20 14:22:08.813 UTC [11166] LOG: background worker "logical replication launcher" (PID 11175) exited with exit code 1
2026-06-20 14:22:08.814 UTC [11170] LOG: shutting down
2026-06-20 14:22:08.979 UTC [11166] LOG: database system is shut down
2026-06-20T14:22:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47165 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:22:09 dist-test-slave-87nc krb5kdc[12149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:22:09 dist-test-slave-87nc krb5kdc[12149](info): set up 2 sockets
Jun 20 14:22:09 dist-test-slave-87nc krb5kdc[12149](info): commencing operation
krb5kdc: starting...
W20260620 14:22:10.156044 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:22:10 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:22:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:22:10.325915 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUQknKY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:22:12.792 UTC [12179] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:22:12.792 UTC [12179] LOG: listening on IPv4 address "127.0.98.20", port 54095
2026-06-20 14:22:12.795 UTC [12179] LOG: listening on Unix socket "/tmp/.s.PGSQL.54095"
2026-06-20 14:22:12.817 UTC [12182] LOG: database system was shut down at 2026-06-20 14:22:11 UTC
2026-06-20 14:22:12.820 UTC [12179] LOG: database system is ready to accept connections
I20260620 14:22:13.815685 392 mini_postgres.cc:96] Postgres bound to 54095
2026-06-20 14:22:13.822 UTC [12190] FATAL: database "slave" does not exist
127.0.98.20:54095 - accepting connections
I20260620 14:22:13.822860 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:22:13.838214 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:22:14.284633 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:22:14.284719 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-admin
2026-06-20 14:22:14,557 [I] DB FLAVOR :POSTGRES
2026-06-20 14:22:14,557 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:22:14,557 [I] Checking connection..
2026-06-20 14:22:14,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:22:14,820 [I] Checking connection passed.
2026-06-20 14:22:14,820 [I] --------- Verifying version history table ---------
2026-06-20 14:22:14,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:22:15,069 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:22:15,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:22:15,333 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:22:15,333 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:22:15,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:22:15.567 UTC [12254] WARNING: there is no transaction in progress
2026-06-20 14:22:15,573 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:22:15,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:22:15,837 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:22:15,837 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:22:15,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:22:16,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:22:16,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-87nc.c.gcp-upstream.internal','N') ;"
2026-06-20 14:22:16,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:22:16,874 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:22:16,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:22:17,110 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:22:17,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:22:17,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:22:17,626 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:22:17,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:22:18.578 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.593 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.609 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.624 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.646 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.665 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.721 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.728 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.742 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.753 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.763 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.777 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.786 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.793 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.800 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:18.806 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 14:22:19,430 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:22:19,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:22:19,681 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:22:19,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:22:19,908 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:22:19,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:22:20,161 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:22:20,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:22:20,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:22:20,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-87nc.c.gcp-upstream.internal';"
2026-06-20 14:22:20,880 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:22:20,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:22:21,136 [I] DB_PATCHES have already been applied
I20260620 14:22:21.141459 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUQknKY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:22:21.141522 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:22:21.144789 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:60883
Jun 20, 2026 2:22:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:22:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:22:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:22:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:22:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:22:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:22:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:22:22.272598 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.450s user 0.001s sys 0.008s
Jun 20, 2026 2:22:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:22:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:22:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:22:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:22:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:22:43.110347 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:22:43 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965363, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:22:43.195672 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:22:43.205679 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:22:44.013346 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:22:44.013429 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-kms
I20260620 14:22:44.013453 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54095
2026-06-20 14:22:44,181 [I] DB FLAVOR :POSTGRES
2026-06-20 14:22:44,181 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:22:44,181 [I] Checking connection
2026-06-20 14:22:44,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:22:44,438 [I] connection success
2026-06-20 14:22:44,438 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:22:44,439 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:22:44,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:22:44,673 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:22:44,673 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:22:44,673 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:22:44,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54095/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:22:44,927 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:22:45.379515 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:22:45.568929 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:22:45.613749 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:22:45 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965363, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:46.028167 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:22:46.028337 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUQknKY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUQknKY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:22:46.028362 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:22:46.033239 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12542
I20260620 14:22:46.033325 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60045
Jun 20, 2026 2:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:22:48.325680 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.130s user 0.019s sys 0.132s
I20260620 14:22:48.325815 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:22:48.325910 392 mini_ranger_kms.cc:401] 127.0.98.20:60045/kms/v1/keys
Jun 20 14:22:48 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:22:48 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:22:49 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:22:51 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:22:51.611943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40037
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40037
--ranger_config_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60045/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:51.732547 12593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:51.732810 12593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:51.732844 12593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:51.736711 12593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:51.736757 12593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:51.736774 12593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:51.736789 12593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:51.736833 12593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:51.742108 12593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60045/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40037
--ranger_config_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:51.743186 12593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:51.744264 12593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:51.753561 12601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:51.753553 12599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:51.753607 12598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:51.754112 12593 server_base.cc:1034] running on GCE node
I20260620 14:22:51.754655 12593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:51.755743 12593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:51.756960 12593 hybrid_clock.cc:648] HybridClock initialized: now 1781965371756943 us; error 36 us; skew 500 ppm
Jun 20 14:22:51 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:51.763746 12593 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:22:51.764678 12593 webserver.cc:460] Webserver started at http://127.0.98.62:39983/ using document root <none> and password file <none>
I20260620 14:22:51.765231 12593 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:51.765308 12593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:51.765512 12593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:22:51 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:51.826897 12593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "05e4d8c4a1e246f8af16d1212e4765f2"
format_stamp: "Formatted at 2026-06-20 14:22:51 on dist-test-slave-87nc"
server_key: "e3a8ae7a375c9398302c2d59c49ab761"
server_key_iv: "c310f6d3bb879e7eb14bd2432478f59c"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:51.827543 12593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "05e4d8c4a1e246f8af16d1212e4765f2"
format_stamp: "Formatted at 2026-06-20 14:22:51 on dist-test-slave-87nc"
server_key: "e3a8ae7a375c9398302c2d59c49ab761"
server_key_iv: "c310f6d3bb879e7eb14bd2432478f59c"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:51.831717 12593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:22:51.849036 12608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:51.850720 12593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:51.850868 12593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "05e4d8c4a1e246f8af16d1212e4765f2"
format_stamp: "Formatted at 2026-06-20 14:22:51 on dist-test-slave-87nc"
server_key: "e3a8ae7a375c9398302c2d59c49ab761"
server_key_iv: "c310f6d3bb879e7eb14bd2432478f59c"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:51.851006 12593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:51.882949 12593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:51.886391 12593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:51.886584 12593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:51.895272 12593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40037
I20260620 14:22:51.895285 12660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40037 every 8 connection(s)
I20260620 14:22:51.896545 12593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:51.899376 12661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:51.900169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 12593
I20260620 14:22:51.900292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:22:51.900625 392 external_mini_cluster.cc:1428] Setting key c98284501d76b9b21a060773eeb09d4b
I20260620 14:22:51.905874 12661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Bootstrap starting.
I20260620 14:22:51.908413 12661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:51.909268 12661 log.cc:824] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:22:51 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:51.911481 12661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: No bootstrap required, opened a new log
I20260620 14:22:51.915767 12661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260620 14:22:51.916044 12661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:51.916152 12661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e4d8c4a1e246f8af16d1212e4765f2, State: Initialized, Role: FOLLOWER
I20260620 14:22:51.916673 12661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260620 14:22:51.916855 12661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:22:51.916937 12661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:22:51.917059 12661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:51.917907 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:51.902028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.0.98.62:40037)
0620 14:22:51.902540 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:51.902562 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:51.902605 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:22:51.903463 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:51.903487 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:51.903954 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:51.904337 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:51.905255 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:51.906914 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:51.908075 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:51.908783 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:51.908947 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:51.911971 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:51.911977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:51.911993 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:51.912022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:51.914667 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:51.915199 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:51.915205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:51.915211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:51.915354 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:51.915660 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:51.915664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:51.915666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:51.916184 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:51.916434 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:51.916687 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 14:22:51.916965 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":126,"threads_started":1}
I20260620 14:22:51.918148 12661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260620 14:22:51.918515 12661 leader_election.cc:302] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05e4d8c4a1e246f8af16d1212e4765f2; no voters:
I20260620 14:22:51.918850 12661 leader_election.cc:288] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:22:51.918952 12666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:51.919214 12666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 1 LEADER]: Becoming Leader. State: Replica: 05e4d8c4a1e246f8af16d1212e4765f2, State: Running, Role: LEADER
I20260620 14:22:51.919575 12666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260620 14:22:51.920182 12661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:51.921070 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } } }
I20260620 14:22:51.921507 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:51.921407 12668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05e4d8c4a1e246f8af16d1212e4765f2. Latest consensus state: current_term: 1 leader_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } } }
I20260620 14:22:51.921844 12668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:51.922152 12676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:51.922400 12661 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:22:51.929042 12676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:22:51.937332 12676 catalog_manager.cc:1347] Generated new cluster ID: 0797416e986b4e748bc5440f93557175
I20260620 14:22:51.937393 12676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:22:51.984908 12676 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:22:51.986339 12676 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:51.996788 12676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Generated new TSK 0
I20260620 14:22:51.997854 12676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:22:52.655833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40037
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:52.797616 12706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:52.797878 12706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:52.797919 12706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:52.803926 12706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:52.804011 12706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:52.804098 12706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:22:52.811910 12706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:52.813236 12706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:52.814585 12706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:52.824782 12713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:52.824784 12716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:52.825232 12714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:52.825301 12706 server_base.cc:1034] running on GCE node
I20260620 14:22:52.825685 12706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:52.826370 12706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:52.827543 12706 hybrid_clock.cc:648] HybridClock initialized: now 1781965372827509 us; error 48 us; skew 500 ppm
Jun 20 14:22:52 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:52.830802 12706 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:22:52.831948 12706 webserver.cc:460] Webserver started at http://127.0.98.1:42935/ using document root <none> and password file <none>
I20260620 14:22:52.832674 12706 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:52.832839 12706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:52.833120 12706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:22:52 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:52.849784 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "7de182856bef4d60b5c9cd25d27357ec"
format_stamp: "Formatted at 2026-06-20 14:22:52 on dist-test-slave-87nc"
server_key: "bba39248547665f2bd89dab1bd66e856"
server_key_iv: "b09a49b2a4a6fc22595f243bc3e13f07"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:52.858605 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7de182856bef4d60b5c9cd25d27357ec"
format_stamp: "Formatted at 2026-06-20 14:22:52 on dist-test-slave-87nc"
server_key: "bba39248547665f2bd89dab1bd66e856"
server_key_iv: "b09a49b2a4a6fc22595f243bc3e13f07"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:52.869839 12706 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.000s
I20260620 14:22:52.908171 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:52.910089 12706 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.000s
I20260620 14:22:52.910249 12706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "7de182856bef4d60b5c9cd25d27357ec"
format_stamp: "Formatted at 2026-06-20 14:22:52 on dist-test-slave-87nc"
server_key: "bba39248547665f2bd89dab1bd66e856"
server_key_iv: "b09a49b2a4a6fc22595f243bc3e13f07"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:52.910431 12706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:52.938452 12706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:52.942414 12706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:52.942677 12706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:52.943444 12706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:52.944587 12706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:52.944715 12706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:52.944847 12706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:52.944934 12706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:22:52 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:52.959074 12706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39735
I20260620 14:22:52.960456 12706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:22:52.961540 12838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39735 every 8 connection(s)
I20260620 14:22:52.967535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 12706
I20260620 14:22:52.967672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:22:52.967996 392 external_mini_cluster.cc:1428] Setting key 9189b8627e5c4fd897a3f09b974cc27c
Jun 20 14:22:52 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:52.981649 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:52.965542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34537 (local address 127.0.98.62:40037)
0620 14:22:52.967917 (+ 2375us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:52.967925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:52.967943 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:22:52.968385 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:52.968389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:52.968446 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:52.968531 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:52.970033 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:52.970936 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:52.972797 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:52.973165 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:52.973279 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:52.976636 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:52.976641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:52.976644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:52.976677 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:52.978764 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:52.979477 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:52.979481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:52.979483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:52.979603 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:52.980123 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:52.980127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:52.980129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:52.980394 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:52.980496 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:52.981266 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 14:22:52.981418 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2276,"thread_start_us":118,"threads_started":1}
I20260620 14:22:52.983605 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:52.964335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40037 (local address 127.0.98.1:34537)
0620 14:22:52.966397 (+ 2062us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:52.966453 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:52.967520 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:52.968835 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:52.968844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:52.969199 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:52.969863 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:52.969883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:52.971113 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:52.971119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:52.972657 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:52.972669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:52.973515 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:52.973519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:52.973727 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:52.974444 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:52.974466 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:52.976387 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:52.978931 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:52.978937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:52.978955 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:52.979326 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:52.979718 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:52.979721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:52.979723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:52.979968 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:52.980586 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:52.980593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:52.980954 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 14:22:52.982134 (+ 1180us) client_negotiation.cc:240] Negotiation successful
0620 14:22:52.982489 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1828,"thread_start_us":140,"threads_started":1}
I20260620 14:22:52.985388 12839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40037
I20260620 14:22:52.985661 12839 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:52.986269 12839 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
I20260620 14:22:52.988305 12625 ts_manager.cc:194] Registered new tserver with Master: 7de182856bef4d60b5c9cd25d27357ec (127.0.98.1:39735)
I20260620 14:22:52.989892 12625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34537
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:22:53.020932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40037
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:53.174544 12845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:53.175366 12845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:53.175518 12845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:53.181805 12845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:53.181988 12845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:53.182135 12845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:22:53.190268 12845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:53.192243 12845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:53.193643 12845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:53.207230 12852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:53.207304 12845 server_base.cc:1034] running on GCE node
W20260620 14:22:53.207691 12851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:53.208348 12854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:53.208881 12845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:53.209642 12845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:53.210794 12845 hybrid_clock.cc:648] HybridClock initialized: now 1781965373210728 us; error 84 us; skew 500 ppm
Jun 20 14:22:53 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:53.214777 12845 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:22:53.215895 12845 webserver.cc:460] Webserver started at http://127.0.98.2:45777/ using document root <none> and password file <none>
I20260620 14:22:53.216755 12845 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:53.216902 12845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:53.217195 12845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:22:53 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:53.236867 12845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b6ae1a0c32d4c618f075dfb646e64c4"
format_stamp: "Formatted at 2026-06-20 14:22:53 on dist-test-slave-87nc"
server_key: "0dec7d87cabe13332bcca585a13b61d6"
server_key_iv: "7674521eabc8b38ed4dbdc4346ba5085"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:53.238349 12845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b6ae1a0c32d4c618f075dfb646e64c4"
format_stamp: "Formatted at 2026-06-20 14:22:53 on dist-test-slave-87nc"
server_key: "0dec7d87cabe13332bcca585a13b61d6"
server_key_iv: "7674521eabc8b38ed4dbdc4346ba5085"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:53.243490 12845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:22:53.262724 12861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:53.265708 12845 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 14:22:53.265949 12845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "1b6ae1a0c32d4c618f075dfb646e64c4"
format_stamp: "Formatted at 2026-06-20 14:22:53 on dist-test-slave-87nc"
server_key: "0dec7d87cabe13332bcca585a13b61d6"
server_key_iv: "7674521eabc8b38ed4dbdc4346ba5085"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:53.266167 12845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:53.296672 12845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:53.300623 12845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:53.300837 12845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:53.301585 12845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:53.302718 12845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:53.302788 12845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:53.302850 12845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:53.302951 12845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 14:22:53.317235 12845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37099
I20260620 14:22:53.317749 12974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37099 every 8 connection(s)
I20260620 14:22:53.326195 12845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:22:53.335397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 12845
I20260620 14:22:53.335538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:22:53.335894 392 external_mini_cluster.cc:1428] Setting key 27c657ade094391901e68faf8b114bfc
Jun 20 14:22:53 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:53.347842 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:53.330425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48251 (local address 127.0.98.62:40037)
0620 14:22:53.330564 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:53.330568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:53.330801 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:22:53.332165 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:53.332170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:53.332243 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:53.332303 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:53.333980 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:53.334838 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:53.336881 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:53.337257 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:53.337347 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:53.342469 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:53.342473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:53.342476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:53.342509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:53.344686 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:53.345680 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:53.345684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:53.345686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:53.345782 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:53.346297 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:53.346301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:53.346303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:53.346585 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:53.346680 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:53.347458 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 14:22:53.347607 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:22:53.349808 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:53.329243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40037 (local address 127.0.98.2:48251)
0620 14:22:53.330633 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:53.330695 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:53.331880 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:53.332725 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:53.332734 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:53.333116 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:53.333790 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:53.333810 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:53.335015 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:53.335018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:53.336721 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:53.336733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:53.337565 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:53.337569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:53.337783 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:53.338638 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:53.338663 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:53.340572 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:53.345124 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:53.345131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:53.345152 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:53.345541 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:53.345935 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:53.345938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:53.345940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:53.346164 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:53.346754 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:53.346761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:53.347146 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 14:22:53.348318 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0620 14:22:53.348714 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":160,"threads_started":1}
I20260620 14:22:53.351655 12975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40037
I20260620 14:22:53.351936 12975 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:53.352582 12975 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
I20260620 14:22:53.353781 12625 ts_manager.cc:194] Registered new tserver with Master: 1b6ae1a0c32d4c618f075dfb646e64c4 (127.0.98.2:37099)
I20260620 14:22:53.354576 12625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48251
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:22:53.384177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40037
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:22:53 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 14:22:53.597061 12980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:53.597594 12980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:53.597703 12980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:53.606995 12980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:53.608807 12980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:53.608982 12980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:22:53.619457 12980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:53.621043 12980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:53.622386 12980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:53.634944 12988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:53.634961 12986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:53.635462 12985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:53.636675 12980 server_base.cc:1034] running on GCE node
I20260620 14:22:53.637102 12980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:53.637877 12980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:53.639034 12980 hybrid_clock.cc:648] HybridClock initialized: now 1781965373638982 us; error 81 us; skew 500 ppm
Jun 20 14:22:53 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:53.642485 12980 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:22:53.643607 12980 webserver.cc:460] Webserver started at http://127.0.98.3:33599/ using document root <none> and password file <none>
I20260620 14:22:53.644325 12980 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:53.644409 12980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:53.644737 12980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:22:53 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:53.660609 12980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "69c7220879844feca5b17d3d78a457a7"
format_stamp: "Formatted at 2026-06-20 14:22:53 on dist-test-slave-87nc"
server_key: "a09693167c21e47f4de83b67e7878500"
server_key_iv: "9b9024e84daccc7664a556d3ebf3846e"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:53.661558 12980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "69c7220879844feca5b17d3d78a457a7"
format_stamp: "Formatted at 2026-06-20 14:22:53 on dist-test-slave-87nc"
server_key: "a09693167c21e47f4de83b67e7878500"
server_key_iv: "9b9024e84daccc7664a556d3ebf3846e"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:53.667378 12980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 14:22:53.698503 12995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:53.700995 12980 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260620 14:22:53.701217 12980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "69c7220879844feca5b17d3d78a457a7"
format_stamp: "Formatted at 2026-06-20 14:22:53 on dist-test-slave-87nc"
server_key: "a09693167c21e47f4de83b67e7878500"
server_key_iv: "9b9024e84daccc7664a556d3ebf3846e"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:53.701452 12980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:53.721907 12980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:53.725554 12980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:53.725908 12980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:53.726619 12980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:53.727687 12980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:53.727834 12980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:53.727977 12980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:53.728050 12980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:53.738847 12980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39787
I20260620 14:22:53.740120 12980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:22:53.746290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 12980
I20260620 14:22:53.746407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:22:53.746726 392 external_mini_cluster.cc:1428] Setting key 8abcb93c560bce5567c2114dcdadaf2a
I20260620 14:22:53.747682 13108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39787 every 8 connection(s)
Jun 20 14:22:53 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:53.767233 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:53.751915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39807 (local address 127.0.98.62:40037)
0620 14:22:53.752054 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:53.752059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:53.752589 (+ 530us) server_negotiation.cc:407] Connection header received
0620 14:22:53.753860 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:53.753863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:53.753923 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:53.753983 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:53.755675 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:53.756513 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:53.758232 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:53.758576 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:53.758659 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:53.762887 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:53.762891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:53.762893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:53.762919 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:53.764781 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:53.765379 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:53.765383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:53.765385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:53.765472 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:53.765904 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:53.765907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:53.765909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:53.766133 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:53.766211 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:53.766947 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 14:22:53.767068 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:22:53.769068 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:53.750823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40037 (local address 127.0.98.3:39807)
0620 14:22:53.752439 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:53.752495 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:53.753589 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:53.754407 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:53.754416 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:53.754791 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:53.755496 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:53.755520 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:53.756674 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:53.756677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:53.758087 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:53.758099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:53.758879 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:53.758883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:53.759083 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:53.760914 (+ 1831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:53.760935 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:53.762684 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:53.764912 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:53.764917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:53.764933 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:53.765254 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:53.765580 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:53.765584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:53.765586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:53.765784 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:53.766301 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:53.766309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:53.766640 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 14:22:53.767640 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0620 14:22:53.768024 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1395,"thread_start_us":158,"threads_started":1}
I20260620 14:22:53.770546 13109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40037
I20260620 14:22:53.770800 13109 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:53.771405 13109 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
I20260620 14:22:53.773072 12625 ts_manager.cc:194] Registered new tserver with Master: 69c7220879844feca5b17d3d78a457a7 (127.0.98.3:39787)
I20260620 14:22:53.773739 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:22:53.773829 12625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39807
I20260620 14:22:53.774488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 12706
I20260620 14:22:53.781795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 12845
I20260620 14:22:53.788775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 12980
I20260620 14:22:53.795159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 12593
I20260620 14:22:53.801627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40037
--webserver_interface=127.0.98.62
--webserver_port=39983
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40037
--ranger_config_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60045/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60045/kms with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:53.912391 13112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:53.912645 13112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:53.912684 13112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:53.916683 13112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:53.916751 13112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:53.916771 13112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:53.916787 13112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:53.916802 13112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:53.922209 13112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60045/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40037
--ranger_config_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:53.923233 13112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:53.924243 13112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:53.932510 13120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:53.932516 13117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:53.932516 13118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:53.932859 13112 server_base.cc:1034] running on GCE node
I20260620 14:22:53.933370 13112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:53.934393 13112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:53.935575 13112 hybrid_clock.cc:648] HybridClock initialized: now 1781965373935562 us; error 30 us; skew 500 ppm
Jun 20 14:22:53 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:53.938179 13112 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:22:53.939054 13112 webserver.cc:460] Webserver started at http://127.0.98.62:39983/ using document root <none> and password file <none>
I20260620 14:22:53.939592 13112 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:53.939683 13112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:22:53 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:53.983708 13112 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:53.986519 13128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:53.987963 13112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:53.988113 13112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "05e4d8c4a1e246f8af16d1212e4765f2"
format_stamp: "Formatted at 2026-06-20 14:22:51 on dist-test-slave-87nc"
server_key: "e3a8ae7a375c9398302c2d59c49ab761"
server_key_iv: "c310f6d3bb879e7eb14bd2432478f59c"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:53.988636 13112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:54.000717 13112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:54.003221 13112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:54.003667 13112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:54.010488 13112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40037
I20260620 14:22:54.010493 13180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40037 every 8 connection(s)
I20260620 14:22:54.011556 13112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:54.015841 13181 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:22:54.017424 13181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Bootstrap starting.
I20260620 14:22:54.018574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 13112
I20260620 14:22:54.019026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39735
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40037
--webserver_port=42935
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60045/kms with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:22:54.034336 13181 log.cc:824] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:54.041551 13181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:22:54.041981 13181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Bootstrap complete.
I20260620 14:22:54.046087 13181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260620 14:22:54.046511 13181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e4d8c4a1e246f8af16d1212e4765f2, State: Initialized, Role: FOLLOWER
I20260620 14:22:54.047006 13181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260620 14:22:54.047163 13181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:22:54.047248 13181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:22:54.047369 13181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:22:54.049414 13181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260620 14:22:54.049752 13181 leader_election.cc:302] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05e4d8c4a1e246f8af16d1212e4765f2; no voters:
I20260620 14:22:54.050078 13181 leader_election.cc:288] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:22:54.050230 13185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:22:54.050534 13185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [term 2 LEADER]: Becoming Leader. State: Replica: 05e4d8c4a1e246f8af16d1212e4765f2, State: Running, Role: LEADER
I20260620 14:22:54.050940 13185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260620 14:22:54.051311 13181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:54.052033 13185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05e4d8c4a1e246f8af16d1212e4765f2. Latest consensus state: current_term: 2 leader_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } } }
I20260620 14:22:54.052173 13185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:54.052634 13192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:54.052583 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e4d8c4a1e246f8af16d1212e4765f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } } }
I20260620 14:22:54.052821 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:54.056437 13192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:22:54.056977 13181 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:22:54.057175 13192 catalog_manager.cc:1259] Loaded cluster ID: 0797416e986b4e748bc5440f93557175
I20260620 14:22:54.057229 13192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:22:54.059619 13192 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:54.060410 13192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Loaded TSK: 0
I20260620 14:22:54.061678 13192 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:22:54.145900 13183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:54.146198 13183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:54.146307 13183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:54.151173 13183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:54.151258 13183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:54.151383 13183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:22:54.157300 13183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42935
--tserver_master_addrs=127.0.98.62:40037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:54.158459 13183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:54.159525 13183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:54.172183 13218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.172297 13221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.172200 13219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.172613 13183 server_base.cc:1034] running on GCE node
I20260620 14:22:54.173059 13183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:54.173769 13183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:54.174973 13183 hybrid_clock.cc:648] HybridClock initialized: now 1781965374174955 us; error 39 us; skew 500 ppm
Jun 20 14:22:54 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:54.178607 13183 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:22:54.179744 13183 webserver.cc:460] Webserver started at http://127.0.98.1:42935/ using document root <none> and password file <none>
I20260620 14:22:54.180526 13183 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:54.180640 13183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:22:54 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:54.200611 13183 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:22:54.204630 13229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.206223 13183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:22:54.206360 13183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "7de182856bef4d60b5c9cd25d27357ec"
format_stamp: "Formatted at 2026-06-20 14:22:52 on dist-test-slave-87nc"
server_key: "bba39248547665f2bd89dab1bd66e856"
server_key_iv: "b09a49b2a4a6fc22595f243bc3e13f07"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:54.207122 13183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:54.222848 13183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:54.226429 13183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:54.226679 13183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:54.227411 13183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:54.228508 13183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:54.228607 13183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.228701 13183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:54.228760 13183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.239264 13183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39735
I20260620 14:22:54.239363 13342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39735 every 8 connection(s)
I20260620 14:22:54.240541 13183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:22:54.248765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 13183
I20260620 14:22:54.249249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37099
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40037
--webserver_port=45777
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60045/kms with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:22:54 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:54.272035 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.246403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40037 (local address 127.0.98.1:33399)
0620 14:22:54.247004 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:54.247057 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:54.248176 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:54.253657 (+ 5481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:54.253666 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:54.254008 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:54.254640 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.254659 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.257596 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.257600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.258952 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.258963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.260136 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.260140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.260344 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.261036 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:54.261055 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:54.262683 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:54.268008 (+ 5325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.268017 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.268035 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.268454 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.268929 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.268935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.268939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.269134 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.269981 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:54.269990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:54.270378 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 14:22:54.270607 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 14:22:54.270986 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":172,"threads_started":1}
I20260620 14:22:54.273903 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.248091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33399 (local address 127.0.98.62:40037)
0620 14:22:54.252353 (+ 4262us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.252375 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.252422 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:22:54.252740 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.252755 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.253188 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.255624 (+ 2436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:54.255959 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.257439 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.259100 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.259960 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.260717 (+ 757us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.262900 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:54.262905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:54.262920 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:54.262948 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:54.267842 (+ 4894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.268622 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.268631 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.268638 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.268801 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.269268 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.269274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.269277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.269828 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:54.272574 (+ 2746us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.272724 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.273020 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4077,"thread_start_us":117,"threads_started":1}
I20260620 14:22:54.276799 13343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40037
I20260620 14:22:54.277045 13343 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.277632 13343 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
W20260620 14:22:54.279960 13343 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40037 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:22:54.280097 13343 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.280270 13343 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
I20260620 14:22:54.281158 13343 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.281311 13343 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
W20260620 14:22:54.282047 13343 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:22:54.401957 13347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:54.402310 13347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:54.402390 13347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:54.408365 13347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:54.408463 13347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:54.408546 13347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:22:54.416419 13347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45777
--tserver_master_addrs=127.0.98.62:40037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:54.417883 13347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:54.419201 13347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:54.430743 13353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.430755 13355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.431229 13347 server_base.cc:1034] running on GCE node
W20260620 14:22:54.431486 13352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.432026 13347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:54.432739 13347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:54.433991 13347 hybrid_clock.cc:648] HybridClock initialized: now 1781965374433958 us; error 52 us; skew 500 ppm
Jun 20 14:22:54 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:54.437141 13347 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:22:54.438011 13347 webserver.cc:460] Webserver started at http://127.0.98.2:45777/ using document root <none> and password file <none>
I20260620 14:22:54.438632 13347 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:54.438753 13347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:22:54 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:54.466874 13347 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:22:54.470300 13363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.472020 13347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:22:54.472163 13347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "1b6ae1a0c32d4c618f075dfb646e64c4"
format_stamp: "Formatted at 2026-06-20 14:22:53 on dist-test-slave-87nc"
server_key: "0dec7d87cabe13332bcca585a13b61d6"
server_key_iv: "7674521eabc8b38ed4dbdc4346ba5085"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:54.472918 13347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:54.500679 13347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:54.504761 13347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:54.505101 13347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:54.505893 13347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:54.507078 13347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:54.507198 13347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.507306 13347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:54.507378 13347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.518276 13347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37099
I20260620 14:22:54.518913 13476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37099 every 8 connection(s)
I20260620 14:22:54.519591 13347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:22:54.528815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 13347
I20260620 14:22:54.529750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39787
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40037
--webserver_port=33599
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60045/kms with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:22:54 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:54.536015 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.522297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43857 (local address 127.0.98.62:40037)
0620 14:22:54.522455 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.522460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.523374 (+ 914us) server_negotiation.cc:407] Connection header received
0620 14:22:54.524541 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.524545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.524599 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.524652 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:54.525959 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.526763 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.527898 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.528245 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.528357 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.531605 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:54.531608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:54.531611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:54.531642 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:54.533353 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.533956 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.533960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.533962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.534056 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.534484 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.534488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.534490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.534730 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:54.534802 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.535692 (+ 890us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.535850 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:22:54.537012 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.522702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40037 (local address 127.0.98.2:43857)
0620 14:22:54.523179 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:54.523230 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:54.524283 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:54.525043 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:54.525050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:54.525326 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:54.525801 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.525814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.526898 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.526900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.527726 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.527734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.528412 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.528415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.528660 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.529395 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:54.529416 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:54.531399 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:54.533452 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.533458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.533472 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.533827 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.534178 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.534181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.534183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.534353 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.534883 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:54.534889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:54.535203 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 14:22:54.535624 (+ 421us) client_negotiation.cc:240] Negotiation successful
0620 14:22:54.535976 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":143,"threads_started":1}
I20260620 14:22:54.538441 13477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40037
I20260620 14:22:54.538733 13477 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.539368 13477 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
W20260620 14:22:54.541193 13477 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40037 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:22:54.541368 13477 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.541587 13477 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
I20260620 14:22:54.542505 13477 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.542711 13477 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
W20260620 14:22:54.543565 13477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:22:54.690347 13480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:54.690740 13480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:54.690809 13480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:54.697286 13480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:54.697423 13480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:54.697542 13480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:22:54.705621 13480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33599
--tserver_master_addrs=127.0.98.62:40037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:54.707149 13480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:54.708599 13480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:54.721215 13487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.722020 13486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.722355 13480 server_base.cc:1034] running on GCE node
W20260620 14:22:54.722527 13489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.723044 13480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:54.723892 13480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:54.725091 13480 hybrid_clock.cc:648] HybridClock initialized: now 1781965374725055 us; error 54 us; skew 500 ppm
Jun 20 14:22:54 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:54.728786 13480 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:22:54.729925 13480 webserver.cc:460] Webserver started at http://127.0.98.3:33599/ using document root <none> and password file <none>
I20260620 14:22:54.730693 13480 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:54.730801 13480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:22:54 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:54.747152 13480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:22:54.749986 13497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.751981 13480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:22:54.752115 13480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "69c7220879844feca5b17d3d78a457a7"
format_stamp: "Formatted at 2026-06-20 14:22:53 on dist-test-slave-87nc"
server_key: "a09693167c21e47f4de83b67e7878500"
server_key_iv: "9b9024e84daccc7664a556d3ebf3846e"
server_key_version: "kuduclusterkey@0"
I20260620 14:22:54.752799 13480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:54.764705 13480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:54.768313 13480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:54.768565 13480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:54.769310 13480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:54.770627 13480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:54.770700 13480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.770758 13480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:54.770839 13480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 14:22:54.784091 13480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39787
I20260620 14:22:54.784093 13610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39787 every 8 connection(s)
I20260620 14:22:54.785477 13480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:22:54.790194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 13480
Jun 20 14:22:54 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:54.802726 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.790921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.98.62:40037)
0620 14:22:54.791949 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.791959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.792270 (+ 311us) server_negotiation.cc:407] Connection header received
0620 14:22:54.792487 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.792491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.792550 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.792634 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:54.793791 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.794538 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.795813 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.796125 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.796203 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.798203 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:54.798207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:54.798210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:54.798238 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:54.800296 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.800885 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.800892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.800897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.801005 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.801446 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.801450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.801452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.801708 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:54.801897 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.802321 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.802463 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":96,"threads_started":1}
I20260620 14:22:54.805519 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.787418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36359 (local address 127.0.98.62:40037)
0620 14:22:54.787568 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.787573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.788636 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 14:22:54.789764 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.789768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.789835 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.789945 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:54.792627 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.793492 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.795243 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.795588 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.795668 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.799901 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:54.799907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:54.799910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:54.799963 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:54.801781 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.802440 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.802444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.802446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.802522 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.803207 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.803212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.803214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.803452 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:54.803547 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.805238 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.805351 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:22:54.806608 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.787959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40037 (local address 127.0.98.3:36359)
0620 14:22:54.788468 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:54.788523 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:54.789535 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:54.791224 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:54.791232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:54.791591 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:54.792425 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.792478 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.793707 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.793714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.795092 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.795104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.795913 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.795917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.796131 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.797099 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:54.797126 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:54.799665 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:54.801915 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.801924 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.801943 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.802324 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.802643 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.802646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.802648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.803083 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.803810 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:54.803817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:54.804781 (+ 964us) client_negotiation.cc:769] Sending connection context
0620 14:22:54.805182 (+ 401us) client_negotiation.cc:240] Negotiation successful
0620 14:22:54.805516 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":149,"threads_started":1}
I20260620 14:22:54.807809 13611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40037
I20260620 14:22:54.808072 13611 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.808635 13611 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
W20260620 14:22:54.810250 13611 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40037 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:22:54.810408 13611 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.810624 13611 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
I20260620 14:22:54.811512 13611 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.811723 13611 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
W20260620 14:22:54.812578 13611 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:22:55 dist-test-slave-87nc krb5kdc[12149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:55.282284 13343 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:55.282588 13343 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
I20260620 14:22:55.283864 13145 ts_manager.cc:194] Registered new tserver with Master: 7de182856bef4d60b5c9cd25d27357ec (127.0.98.1:39735)
I20260620 14:22:55.285593 13145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33399
I20260620 14:22:55.543849 13477 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:55.544147 13477 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
I20260620 14:22:55.544872 13145 ts_manager.cc:194] Registered new tserver with Master: 1b6ae1a0c32d4c618f075dfb646e64c4 (127.0.98.2:37099)
I20260620 14:22:55.545653 13145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43857
Jun 20 14:22:55 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:55.812844 13611 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:55.813172 13611 heartbeater.cc:511] Master 127.0.98.62:40037 requested a full tablet report, sending...
I20260620 14:22:55.813993 13145 ts_manager.cc:194] Registered new tserver with Master: 69c7220879844feca5b17d3d78a457a7 (127.0.98.3:39787)
I20260620 14:22:55.814529 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:22:55.814715 13145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36359
Jun 20 14:22:55 dist-test-slave-87nc krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:22:55.831318 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.816246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.0.98.62:40037)
0620 14:22:55.816457 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.816461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.816691 (+ 230us) server_negotiation.cc:407] Connection header received
0620 14:22:55.816862 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.816865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.816910 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.821710 (+ 4800us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:55.822770 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.823595 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.824757 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.825127 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.825206 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.827044 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:55.827048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:55.827050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:55.827090 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:55.829399 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:55.829883 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:55.829886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:55.829888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:55.829975 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:55.830289 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:55.830318 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:55.830320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:55.830529 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:55.830648 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.831000 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.831140 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20260620 14:22:55.836020 13145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:22:55.840374 13145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:22:55.875716 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.862453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37099 (local address 127.0.0.1:52762)
0620 14:22:55.864444 (+ 1991us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.864500 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.864795 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.869297 (+ 4502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.869300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.869317 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.869399 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.869405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.871119 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.871122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.873823 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.873834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.875139 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.875142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.875292 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.875335 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.875435 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.875550 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1815,"mutex_wait_us":31,"thread_start_us":125,"threads_started":1}
I20260620 14:22:55.876375 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.863846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.0.98.2:37099)
0620 14:22:55.866513 (+ 2667us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.866522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.866547 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:22:55.866630 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.866638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.867549 (+ 911us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.868086 (+ 537us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.869528 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.871001 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.873970 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.874827 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.874921 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.875075 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.876129 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.876238 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":127,"threads_started":1}
I20260620 14:22:55.877784 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.863394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39735 (local address 127.0.0.1:43054)
0620 14:22:55.866146 (+ 2752us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.866159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.866951 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.870392 (+ 3441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.870397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.870416 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.870791 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.870803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.872433 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.872436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.874688 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.874699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.877179 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.877386 (+ 207us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.877480 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.877510 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.877583 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.877656 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2679,"thread_start_us":107,"threads_started":1}
I20260620 14:22:55.879914 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.863549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.0.98.1:39735)
0620 14:22:55.864173 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.864182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.866248 (+ 2066us) server_negotiation.cc:407] Connection header received
0620 14:22:55.867151 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.867158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.869010 (+ 1852us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.869146 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.870934 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.872052 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.876018 (+ 3966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.876841 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.876921 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.877080 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.878029 (+ 949us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.878131 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":60,"threads_started":1}
I20260620 14:22:55.880542 13402 tablet_service.cc:1467] Processing CreateTablet for tablet ce17c5a542774269a729c580ec6c54cc (DEFAULT_TABLE table=test-table [id=f4bf53ffd42a4bae975cbe66a3ebc242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:55.881460 13402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce17c5a542774269a729c580ec6c54cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:55.885828 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.862683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.0.98.3:39787)
0620 14:22:55.863078 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.863085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.868018 (+ 4933us) server_negotiation.cc:407] Connection header received
0620 14:22:55.868099 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.868104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.868302 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.868479 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.868796 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.869935 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.883929 (+ 13994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.884796 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.884911 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.885052 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.885546 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.885664 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":72,"threads_started":1}
I20260620 14:22:55.886276 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.862453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39787 (local address 127.0.0.1:50346)
0620 14:22:55.867850 (+ 5397us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.867862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.867949 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.868475 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.868481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.868503 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.868655 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.868666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.873558 (+ 4892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.873561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.875870 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.875884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.885234 (+ 9350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.885239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.885350 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.885377 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.886043 (+ 666us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.886140 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5325,"mutex_wait_us":1653,"thread_start_us":81,"threads_started":1}
I20260620 14:22:55.886915 13277 tablet_service.cc:1467] Processing CreateTablet for tablet ce17c5a542774269a729c580ec6c54cc (DEFAULT_TABLE table=test-table [id=f4bf53ffd42a4bae975cbe66a3ebc242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:55.888105 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce17c5a542774269a729c580ec6c54cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:55.889196 13545 tablet_service.cc:1467] Processing CreateTablet for tablet ce17c5a542774269a729c580ec6c54cc (DEFAULT_TABLE table=test-table [id=f4bf53ffd42a4bae975cbe66a3ebc242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:55.890261 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce17c5a542774269a729c580ec6c54cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:55.893167 13646 tablet_bootstrap.cc:492] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4: Bootstrap starting.
I20260620 14:22:55.896308 13646 tablet_bootstrap.cc:654] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:55.897265 13646 log.cc:824] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:55.899501 13646 tablet_bootstrap.cc:492] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4: No bootstrap required, opened a new log
I20260620 14:22:55.899703 13646 ts_tablet_manager.cc:1397] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:22:55.900645 13647 tablet_bootstrap.cc:492] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec: Bootstrap starting.
I20260620 14:22:55.902822 13647 tablet_bootstrap.cc:654] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:55.903235 13646 raft_consensus.cc:348] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } }
I20260620 14:22:55.903450 13646 raft_consensus.cc:374] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:55.903482 13646 raft_consensus.cc:729] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6ae1a0c32d4c618f075dfb646e64c4, State: Initialized, Role: FOLLOWER
I20260620 14:22:55.903591 13647 log.cc:824] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:55.904625 13646 consensus_queue.cc:260] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } }
I20260620 14:22:55.905617 13646 ts_tablet_manager.cc:1428] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:22:55.905692 13647 tablet_bootstrap.cc:492] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec: No bootstrap required, opened a new log
I20260620 14:22:55.905876 13647 ts_tablet_manager.cc:1397] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:22:55.907966 13477 heartbeater.cc:503] Master 127.0.98.62:40037 was elected leader, sending a full tablet report...
I20260620 14:22:55.908193 13649 tablet_bootstrap.cc:492] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: Bootstrap starting.
I20260620 14:22:55.909915 13647 raft_consensus.cc:348] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } }
I20260620 14:22:55.910179 13647 raft_consensus.cc:374] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:55.910225 13647 raft_consensus.cc:729] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de182856bef4d60b5c9cd25d27357ec, State: Initialized, Role: FOLLOWER
I20260620 14:22:55.910539 13649 tablet_bootstrap.cc:654] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:55.910691 13647 consensus_queue.cc:260] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } }
I20260620 14:22:55.911355 13649 log.cc:824] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:55.911747 13647 ts_tablet_manager.cc:1428] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:22:55.912103 13343 heartbeater.cc:503] Master 127.0.98.62:40037 was elected leader, sending a full tablet report...
I20260620 14:22:55.918061 13649 tablet_bootstrap.cc:492] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: No bootstrap required, opened a new log
I20260620 14:22:55.918358 13649 ts_tablet_manager.cc:1397] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260620 14:22:55.922740 13649 raft_consensus.cc:348] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } }
I20260620 14:22:55.923106 13649 raft_consensus.cc:374] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:55.923216 13649 raft_consensus.cc:729] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c7220879844feca5b17d3d78a457a7, State: Initialized, Role: FOLLOWER
I20260620 14:22:55.924685 13649 consensus_queue.cc:260] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } }
I20260620 14:22:55.925901 13649 ts_tablet_manager.cc:1428] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:22:55.928674 13611 heartbeater.cc:503] Master 127.0.98.62:40037 was elected leader, sending a full tablet report...
I20260620 14:22:55.990593 13652 raft_consensus.cc:482] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:55.990844 13652 raft_consensus.cc:504] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } }
I20260620 14:22:55.992415 13652 leader_election.cc:288] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b6ae1a0c32d4c618f075dfb646e64c4 (127.0.98.2:37099), 69c7220879844feca5b17d3d78a457a7 (127.0.98.3:39787)
W20260620 14:22:56.003485 13344 tablet.cc:2369] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:22:56.011547 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.993208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47613 (local address 127.0.98.3:39787)
0620 14:22:55.993364 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.993369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.994654 (+ 1285us) server_negotiation.cc:407] Connection header received
0620 14:22:55.995918 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.995922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.995976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.996024 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.999238 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:56.000049 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.005983 (+ 5934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:56.006729 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.006817 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:56.006889 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:56.011310 (+ 4421us) server_negotiation.cc:299] Negotiation successful
0620 14:22:56.011416 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:22:56.012413 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.993401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54241 (local address 127.0.98.2:37099)
0620 14:22:55.993516 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.993520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.994852 (+ 1332us) server_negotiation.cc:407] Connection header received
0620 14:22:55.995174 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.995178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.995235 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.995395 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.996986 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.997789 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.008224 (+ 10435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:56.008885 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.008969 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:56.009029 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:56.012179 (+ 3150us) server_negotiation.cc:299] Negotiation successful
0620 14:22:56.012294 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:22:56.013001 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.992981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37099 (local address 127.0.98.1:54241)
0620 14:22:55.994776 (+ 1795us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.994787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.994935 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.995584 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.995588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.995606 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.995748 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.995755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:56.000212 (+ 4457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.000215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:56.001605 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:56.001617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:56.011020 (+ 9403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.011024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:56.011148 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:56.011172 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:22:56.012753 (+ 1581us) client_negotiation.cc:240] Negotiation successful
0620 14:22:56.012836 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1741,"thread_start_us":114,"threads_started":1}
I20260620 14:22:56.013567 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.992731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39787 (local address 127.0.98.1:47613)
0620 14:22:55.994547 (+ 1816us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.994563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.995035 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.996673 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.996677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.996700 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.996832 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.996838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:56.003530 (+ 6692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.003533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:56.004786 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:56.004798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:56.007915 (+ 3117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.007919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:56.008028 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:56.008054 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:22:56.012008 (+ 3954us) client_negotiation.cc:240] Negotiation successful
0620 14:22:56.013441 (+ 1433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1741,"thread_start_us":157,"threads_started":1}
I20260620 14:22:56.014878 13548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce17c5a542774269a729c580ec6c54cc" candidate_uuid: "7de182856bef4d60b5c9cd25d27357ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69c7220879844feca5b17d3d78a457a7" is_pre_election: true
I20260620 14:22:56.015228 13548 raft_consensus.cc:2454] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7de182856bef4d60b5c9cd25d27357ec in term 0.
I20260620 14:22:56.015718 13232 leader_election.cc:302] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69c7220879844feca5b17d3d78a457a7, 7de182856bef4d60b5c9cd25d27357ec; no voters:
I20260620 14:22:56.016150 13652 raft_consensus.cc:2781] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:22:56.016234 13652 raft_consensus.cc:482] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:22:56.016274 13652 raft_consensus.cc:3037] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:56.016376 13431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce17c5a542774269a729c580ec6c54cc" candidate_uuid: "7de182856bef4d60b5c9cd25d27357ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" is_pre_election: true
I20260620 14:22:56.016666 13431 raft_consensus.cc:2454] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7de182856bef4d60b5c9cd25d27357ec in term 0.
I20260620 14:22:56.017583 13652 raft_consensus.cc:504] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } }
I20260620 14:22:56.017939 13652 leader_election.cc:288] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [CANDIDATE]: Term 1 election: Requested vote from peers 1b6ae1a0c32d4c618f075dfb646e64c4 (127.0.98.2:37099), 69c7220879844feca5b17d3d78a457a7 (127.0.98.3:39787)
I20260620 14:22:56.018713 13548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce17c5a542774269a729c580ec6c54cc" candidate_uuid: "7de182856bef4d60b5c9cd25d27357ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69c7220879844feca5b17d3d78a457a7"
I20260620 14:22:56.018833 13548 raft_consensus.cc:3037] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:56.019944 13431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce17c5a542774269a729c580ec6c54cc" candidate_uuid: "7de182856bef4d60b5c9cd25d27357ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4"
I20260620 14:22:56.020058 13431 raft_consensus.cc:3037] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:56.021271 13431 raft_consensus.cc:2454] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de182856bef4d60b5c9cd25d27357ec in term 1.
I20260620 14:22:56.021643 13233 leader_election.cc:302] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6ae1a0c32d4c618f075dfb646e64c4, 7de182856bef4d60b5c9cd25d27357ec; no voters:
W20260620 14:22:56.021800 13478 tablet.cc:2369] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:22:56.022202 13548 raft_consensus.cc:2454] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de182856bef4d60b5c9cd25d27357ec in term 1.
I20260620 14:22:56.022591 13652 raft_consensus.cc:2781] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:56.022815 13652 raft_consensus.cc:686] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [term 1 LEADER]: Becoming Leader. State: Replica: 7de182856bef4d60b5c9cd25d27357ec, State: Running, Role: LEADER
I20260620 14:22:56.023168 13652 consensus_queue.cc:237] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } }
I20260620 14:22:56.026592 13145 catalog_manager.cc:5495] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 7de182856bef4d60b5c9cd25d27357ec (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7de182856bef4d60b5c9cd25d27357ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7de182856bef4d60b5c9cd25d27357ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } health_report { overall_health: UNKNOWN } } }
W20260620 14:22:56.038959 13612 tablet.cc:2369] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:22:56.074698 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:56.069459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.0.98.1:39735)
0620 14:22:56.069584 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:56.069588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:56.069603 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:22:56.069700 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:56.069703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:56.069770 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:56.069883 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:22:56.070321 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:56.071138 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.072375 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:56.072718 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:56.072822 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:56.073266 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:22:56.073857 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:22:56.073973 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:56.074460 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 14:22:56.074523 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:22:56.092950 13548 raft_consensus.cc:1261] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [term 1 FOLLOWER]: Refusing update from remote peer 7de182856bef4d60b5c9cd25d27357ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:56.094043 13431 raft_consensus.cc:1261] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [term 1 FOLLOWER]: Refusing update from remote peer 7de182856bef4d60b5c9cd25d27357ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:56.094955 13657 consensus_queue.cc:1035] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b6ae1a0c32d4c618f075dfb646e64c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:56.095975 13657 consensus_queue.cc:1035] T ce17c5a542774269a729c580ec6c54cc P 7de182856bef4d60b5c9cd25d27357ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "69c7220879844feca5b17d3d78a457a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:56.125324 13662 mvcc.cc:205] Tried to move back new op lower bound from 7298930180465106944 to 7298930180194123776. Current Snapshot: MvccSnapshot[applied={T|T < 7298930180465106944}]
I20260620 14:22:56.131808 13145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:56.131989 13145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:56.135963 13145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05e4d8c4a1e246f8af16d1212e4765f2: Sending DeleteTablet for 3 replicas of tablet ce17c5a542774269a729c580ec6c54cc
I20260620 14:22:56.137738 13402 tablet_service.cc:1514] Processing DeleteTablet for tablet ce17c5a542774269a729c580ec6c54cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52762
I20260620 14:22:56.138291 13545 tablet_service.cc:1514] Processing DeleteTablet for tablet ce17c5a542774269a729c580ec6c54cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50346
I20260620 14:22:56.139243 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet ce17c5a542774269a729c580ec6c54cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43054
I20260620 14:22:56.139981 13661 mvcc.cc:205] Tried to move back new op lower bound from 7298930180465106944 to 7298930180194123776. Current Snapshot: MvccSnapshot[applied={T|T < 7298930180465106944}]
I20260620 14:22:56.140317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 13183
I20260620 14:22:56.149675 13669 tablet_replica.cc:330] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4: stopping tablet replica
I20260620 14:22:56.150074 13669 raft_consensus.cc:2229] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:56.150444 13669 raft_consensus.cc:2258] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:56.150997 13670 tablet_replica.cc:330] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: stopping tablet replica
I20260620 14:22:56.151386 13670 raft_consensus.cc:2229] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:56.151752 13670 raft_consensus.cc:2258] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:56.154105 13670 ts_tablet_manager.cc:1905] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:56.156029 13669 ts_tablet_manager.cc:1905] T ce17c5a542774269a729c580ec6c54cc P 1b6ae1a0c32d4c618f075dfb646e64c4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:22:56.157282 13132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39735 (error 108)
I20260620 14:22:56.157449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 13347
I20260620 14:22:56.160391 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:56.160049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39735 (local address 127.0.0.1:43068)
0620 14:22:56.160210 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:56.160293 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 14:22:56.160699 13132 catalog_manager.cc:4584] TS 7de182856bef4d60b5c9cd25d27357ec (127.0.98.1:39735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce17c5a542774269a729c580ec6c54cc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39735: connect: Connection refused (error 111)
I20260620 14:22:56.161043 13670 ts_tablet_manager.cc:1918] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:56.161180 13670 log.cc:1196] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/ce17c5a542774269a729c580ec6c54cc
I20260620 14:22:56.161514 13670 ts_tablet_manager.cc:1939] T ce17c5a542774269a729c580ec6c54cc P 69c7220879844feca5b17d3d78a457a7: Deleting consensus metadata
I20260620 14:22:56.162788 13131 catalog_manager.cc:4841] TS 69c7220879844feca5b17d3d78a457a7 (127.0.98.3:39787): tablet ce17c5a542774269a729c580ec6c54cc (table test-table [id=f4bf53ffd42a4bae975cbe66a3ebc242]) successfully deleted
I20260620 14:22:56.164845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 13480
I20260620 14:22:56.169133 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:56.168791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37099 (local address 127.0.0.1:52768)
0620 14:22:56.168977 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:56.169033 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 14:22:56.174803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 13112
I20260620 14:22:56.181077 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:22:56.526443 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:22:56.526579 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:22:56.861796 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:22:56.862 UTC [12179] LOG: received smart shutdown request
2026-06-20 14:22:56.865 UTC [12179] LOG: background worker "logical replication launcher" (PID 12188) exited with exit code 1
2026-06-20 14:22:56.866 UTC [12183] LOG: shutting down
2026-06-20 14:22:57.027 UTC [12179] LOG: database system is shut down
2026-06-20T14:22:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48050 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:22:57 dist-test-slave-87nc krb5kdc[13684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:22:57 dist-test-slave-87nc krb5kdc[13684](info): set up 2 sockets
Jun 20 14:22:57 dist-test-slave-87nc krb5kdc[13684](info): commencing operation
krb5kdc: starting...
W20260620 14:22:58.283963 392 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:22:58 dist-test-slave-87nc krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:22:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:22:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:22:58.406253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41843
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41843,127.0.98.61:36601,127.0.98.60:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:58.520900 13697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:58.521191 13697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:58.521232 13697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:58.525385 13697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:58.525465 13697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:58.525484 13697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:58.525502 13697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:58.525517 13697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:58.531173 13697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41843,127.0.98.61:36601,127.0.98.60:39989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:58.532392 13697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:58.533416 13697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:58.542128 13705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.542140 13703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.542068 13702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.542433 13697 server_base.cc:1034] running on GCE node
I20260620 14:22:58.543061 13697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:58.544238 13697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:58.545405 13697 hybrid_clock.cc:648] HybridClock initialized: now 1781965378545365 us; error 52 us; skew 500 ppm
Jun 20 14:22:58 dist-test-slave-87nc krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:58.548249 13697 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:22:58.549078 13697 webserver.cc:460] Webserver started at http://127.0.98.62:44973/ using document root <none> and password file <none>
I20260620 14:22:58.549587 13697 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:58.549645 13697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:58.549816 13697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:58.551844 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "f29da3bda5384342aa006a03451fb55b"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-87nc"
server_key: "ea11d933909ccd59c9fc55fd7b7fdb4a"
server_key_iv: "16e830b4e3a04e7053472edd20100261"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.552366 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "f29da3bda5384342aa006a03451fb55b"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-87nc"
server_key: "ea11d933909ccd59c9fc55fd7b7fdb4a"
server_key_iv: "16e830b4e3a04e7053472edd20100261"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.556053 13697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:58.558531 13712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.559916 13697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:58.560060 13697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "f29da3bda5384342aa006a03451fb55b"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-87nc"
server_key: "ea11d933909ccd59c9fc55fd7b7fdb4a"
server_key_iv: "16e830b4e3a04e7053472edd20100261"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.560181 13697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:58.573655 13697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:58.576617 13697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:58.576823 13697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:58.584021 13697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41843
I20260620 14:22:58.584045 13764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41843 every 8 connection(s)
I20260620 14:22:58.585043 13697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:58.587742 13765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:58.591487 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:22:58.592703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 13697
I20260620 14:22:58.592828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:22:58.593123 392 external_mini_cluster.cc:1428] Setting key c03bf319bab6e773e3d67fd75155f160
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 14:22:58 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:58.609457 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.593929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.0.98.62:41843)
0620 14:22:58.594482 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.594495 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.594571 (+ 76us) server_negotiation.cc:407] Connection header received
0620 14:22:58.595663 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.595667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.595922 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.596110 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:58.596959 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.597826 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.598850 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.599229 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.599398 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.604353 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:58.604364 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:58.604382 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:58.604419 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:58.606199 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:58.606782 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:58.606787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:58.606789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:58.606894 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:58.607262 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:58.607264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:58.607266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:58.607722 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:58.607908 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.608383 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.608602 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":150,"threads_started":1}
I20260620 14:22:58.609454 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.593928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41843 (local address 127.0.0.1:48274)
0620 14:22:58.594417 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.594470 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.595345 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.596110 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.596115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.596567 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:58.596812 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.596825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.597958 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.597962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.598728 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.598737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.599383 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.599387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.599493 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.600168 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:58.600184 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:58.604176 (+ 3992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:58.606314 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:58.606326 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:58.606340 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:58.606667 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:58.606993 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:58.606999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:58.607004 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:58.607161 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:58.607898 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:58.607908 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:58.608092 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.608323 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.608679 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":6,"thread_start_us":146,"threads_started":1}
I20260620 14:22:58.612170 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:22:58.612874 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.612657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36601 (local address 127.0.0.1:54542)
0620 14:22:58.612752 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.612820 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:22:58.613061 13713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
I20260620 14:22:58.613968 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.613728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36601 (local address 127.0.0.1:54554)
0620 14:22:58.613879 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.613920 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 14:22:58.614100 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36601: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:22:58.619134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36601
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41843,127.0.98.61:36601,127.0.98.60:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:22:58.663367 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } attempt: 1
I20260620 14:22:58.664419 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.664060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36601 (local address 127.0.0.1:54564)
0620 14:22:58.664257 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.664328 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:22:58.665319 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.665064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36601 (local address 127.0.0.1:54576)
0620 14:22:58.665211 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.665261 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 14:22:58.665490 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36601: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
I20260620 14:22:58.733682 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } attempt: 2
W20260620 14:22:58.734143 13772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:58.734452 13772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:58.734565 13772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:22:58.734652 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.734283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36601 (local address 127.0.0.1:54588)
0620 14:22:58.734431 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.734527 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:22:58.735406 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.735174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36601 (local address 127.0.0.1:54598)
0620 14:22:58.735284 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.735329 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:22:58.735595 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36601: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36601: connect: Connection refused (error 111)
W20260620 14:22:58.739161 13772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:58.739250 13772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:58.739287 13772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:58.739342 13772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:58.739378 13772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:58.745414 13772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41843,127.0.98.61:36601,127.0.98.60:39989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:58.746551 13772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:58.747607 13772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:58.756457 13777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.756484 13778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.756503 13772 server_base.cc:1034] running on GCE node
W20260620 14:22:58.756457 13780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.757361 13772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:58.758477 13772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:58.759646 13772 hybrid_clock.cc:648] HybridClock initialized: now 1781965378759619 us; error 40 us; skew 500 ppm
Jun 20 14:22:58 dist-test-slave-87nc krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:58.762568 13772 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:22:58.763530 13772 webserver.cc:460] Webserver started at http://127.0.98.61:41357/ using document root <none> and password file <none>
I20260620 14:22:58.764180 13772 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:58.764245 13772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:58.764477 13772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:58.766601 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/data/instance:
uuid: "cd97cea530a743b0a2c700546381c1ff"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-87nc"
server_key: "4426a319fa414c023463c3782ca3329b"
server_key_iv: "b0cda29c99c7259832ed450943585a0c"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.767166 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/wal/instance:
uuid: "cd97cea530a743b0a2c700546381c1ff"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-87nc"
server_key: "4426a319fa414c023463c3782ca3329b"
server_key_iv: "b0cda29c99c7259832ed450943585a0c"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.771107 13772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 14:22:58.773645 13787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.774966 13772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:22:58.775226 13772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/wal
uuid: "cd97cea530a743b0a2c700546381c1ff"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-87nc"
server_key: "4426a319fa414c023463c3782ca3329b"
server_key_iv: "b0cda29c99c7259832ed450943585a0c"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.775352 13772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:58.796530 13772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:58.800000 13772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:58.800251 13772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:58.807401 13772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36601
I20260620 14:22:58.807428 13839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36601 every 8 connection(s)
I20260620 14:22:58.808363 13772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:22:58.811054 13840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:58.814888 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:22:58.815495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 13772
I20260620 14:22:58.815625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-1/wal/instance
I20260620 14:22:58.815964 392 external_mini_cluster.cc:1428] Setting key 6e0c8933d06b66281e49e952068918b1
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 14:22:58.826756 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } attempt: 3
Jun 20 14:22:58 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:58.832765 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.816948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.98.62:41843)
0620 14:22:58.817108 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.817113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.818031 (+ 918us) server_negotiation.cc:407] Connection header received
0620 14:22:58.818962 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.818968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.819036 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.819145 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:58.820396 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.821333 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.822464 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.822873 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.822995 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.827821 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:58.827828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:58.827834 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:58.827867 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:58.830110 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:58.830777 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:58.830780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:58.830782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:58.830872 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:58.831322 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:58.831325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:58.831328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:58.831545 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:58.831646 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.832428 (+ 782us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.832570 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:22:58 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:22:58.834067 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.817351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41843 (local address 127.0.0.1:48290)
0620 14:22:58.817841 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.817894 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.818752 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.819307 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.819313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.819645 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:58.820234 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.820248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.821474 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.821476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.822336 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.822345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.823029 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.823032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.823163 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.823879 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:58.823895 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:58.827618 (+ 3723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:58.830231 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:58.830240 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:58.830259 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:58.830667 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:58.830982 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:58.830988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:58.830993 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:58.831213 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:58.831671 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:58.831679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:58.832107 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.832360 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.832737 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":144,"threads_started":1}
I20260620 14:22:58.835928 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:22:58.838254 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.827261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36601 (local address 127.0.0.1:54608)
0620 14:22:58.827477 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.827497 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.827590 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.828393 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.828399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.828602 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:58.828753 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.828761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.830475 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.830478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.831215 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.831224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.832103 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.832106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.832222 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.832710 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:58.832724 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:58.834038 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:58.836264 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:58.836268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:58.836272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:58.836471 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:58.837023 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:58.837026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:58.837028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:58.837125 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:58.837698 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:58.837701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:58.837833 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.837955 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.838091 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 14:22:58.838394 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.827399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.0.98.61:36601)
0620 14:22:58.827889 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.827899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.827919 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:22:58.827956 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.827960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.828233 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.828392 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:58.828905 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.829976 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.831386 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.831972 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.832089 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.834180 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:58.834190 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:58.834206 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:58.834231 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:58.836147 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:58.836586 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:58.836592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:58.836593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:58.836681 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:58.837240 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:58.837243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:58.837245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:58.837579 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:58.837702 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.838086 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.838243 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":132,"threads_started":1}
I20260620 14:22:58.841042 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:22:58 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:22:58.841830 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.841539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52276)
0620 14:22:58.841677 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.841737 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:22:58.842581 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.842333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52292)
0620 14:22:58.842488 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.842531 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 14:22:58.842968 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39989: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
I20260620 14:22:58.845477 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.836782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36601 (local address 127.0.0.1:54614)
0620 14:22:58.836898 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.836910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.836973 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.837492 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.837498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.837654 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:58.837755 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.837761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.838922 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.838928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.839500 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.839509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.840056 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.840059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.840158 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.840672 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:58.840685 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:58.842163 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:58.844058 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:58.844064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:58.844067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:58.844251 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:58.844532 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:58.844535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:58.844537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:58.844615 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:58.844978 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:58.844981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:58.845059 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.845171 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.845320 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:22:58.845590 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.836839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.0.98.61:36601)
0620 14:22:58.837142 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.837152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.837167 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:22:58.837204 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.837209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.837253 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.837338 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:58.837874 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.838822 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.839625 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.839943 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.840037 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.842319 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:58.842323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:58.842326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:58.842350 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:58.843956 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:58.844354 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:58.844359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:58.844361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:58.844438 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:58.844700 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:58.844705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:58.844707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:58.844883 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:58.844990 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.845222 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.845417 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:22:58.846297 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:22:58.847040 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.846799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52302)
0620 14:22:58.846920 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.846986 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:22:58.847031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39989
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41843,127.0.98.61:36601,127.0.98.60:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:58.847247 13790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
I20260620 14:22:58.848166 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.847949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52316)
0620 14:22:58.848055 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.848099 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:22:58.848330 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39989: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
I20260620 14:22:58.874150 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } attempt: 1
I20260620 14:22:58.875046 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.874703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52320)
0620 14:22:58.874874 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.874951 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 14:22:58.875857 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.875589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52330)
0620 14:22:58.875709 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.875788 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:22:58.876030 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39989: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
I20260620 14:22:58.897625 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } attempt: 1
I20260620 14:22:58.898488 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.898181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52336)
0620 14:22:58.898346 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.898414 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 14:22:58.899295 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.899076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52344)
0620 14:22:58.899186 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.899233 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:22:58.899474 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39989: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
I20260620 14:22:58.951225 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } attempt: 2
I20260620 14:22:58.952167 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.951825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52346)
0620 14:22:58.951996 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.952070 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:22:58.953362 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.953130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52352)
0620 14:22:58.953246 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.953299 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:22:58.953496 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39989: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
W20260620 14:22:58.960440 13848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:58.960700 13848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:58.960741 13848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:58.964980 13848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:58.965046 13848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:58.965067 13848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:58.965085 13848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:58.965101 13848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:58.967630 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } attempt: 2
I20260620 14:22:58.968470 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.968181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52364)
0620 14:22:58.968335 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.968397 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:22:58.969151 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.968964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52370)
0620 14:22:58.969057 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.969100 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:22:58.969298 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39989: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39989: connect: Connection refused (error 111)
I20260620 14:22:58.971004 13848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41843,127.0.98.61:36601,127.0.98.60:39989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:22:58.972177 13848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:58.973330 13848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:58.981962 13854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.981961 13853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.982028 13848 server_base.cc:1034] running on GCE node
W20260620 14:22:58.981978 13856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.982872 13848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:58.984066 13848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:58.985265 13848 hybrid_clock.cc:648] HybridClock initialized: now 1781965378985255 us; error 36 us; skew 500 ppm
Jun 20 14:22:58 dist-test-slave-87nc krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:58.988126 13848 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:22:58.989089 13848 webserver.cc:460] Webserver started at http://127.0.98.60:37247/ using document root <none> and password file <none>
I20260620 14:22:58.989655 13848 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:58.989749 13848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:58.989980 13848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:58.992061 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/data/instance:
uuid: "c2ff2dfbc27348de8dae2cd871ef0496"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-87nc"
server_key: "80d9daf513a27c806daeb64e8e3ace41"
server_key_iv: "0f97dd45231c8c926b4f9d9eb2aa1475"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.992631 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/wal/instance:
uuid: "c2ff2dfbc27348de8dae2cd871ef0496"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-87nc"
server_key: "80d9daf513a27c806daeb64e8e3ace41"
server_key_iv: "0f97dd45231c8c926b4f9d9eb2aa1475"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.996300 13848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:58.998772 13863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:59.000087 13848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:22:59.000236 13848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/wal
uuid: "c2ff2dfbc27348de8dae2cd871ef0496"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-87nc"
server_key: "80d9daf513a27c806daeb64e8e3ace41"
server_key_iv: "0f97dd45231c8c926b4f9d9eb2aa1475"
server_key_version: "encryptionkey@0"
I20260620 14:22:59.000355 13848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:59.012183 13848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:59.015606 13848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:59.015872 13848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:59.022792 13848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39989
I20260620 14:22:59.022821 13915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39989 every 8 connection(s)
I20260620 14:22:59.023808 13848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:22:59.026422 13916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:59.030141 13916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:22:59.033758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 13848
I20260620 14:22:59.033865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965214882667-392-0/minicluster-data/master-2/wal/instance
I20260620 14:22:59.034135 392 external_mini_cluster.cc:1428] Setting key aaf3f0df398856aa47849c64a410e46b
Jun 20 14:22:59 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:22:59 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:22:59.044762 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.032068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.0.98.62:41843)
0620 14:22:59.032236 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.032241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.033094 (+ 853us) server_negotiation.cc:407] Connection header received
0620 14:22:59.033982 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.033986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.034045 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.034150 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.035599 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.036174 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.037285 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.037503 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.037619 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.040517 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.040523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.040529 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.040566 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.042419 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.042997 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.043002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.043004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.043097 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.043470 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.043475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.043477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.043709 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.043877 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.044426 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.044570 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:22:59.045720 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.032477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41843 (local address 127.0.0.1:48292)
0620 14:22:59.032930 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:59.032972 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:59.033726 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:59.034337 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:59.034344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:59.034819 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:59.035459 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.035475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.036319 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.036323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.037149 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.037159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.037653 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.037657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.037841 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.038458 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:59.038475 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:59.040366 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:59.042545 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.042555 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.042567 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.042884 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.043208 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.043211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.043213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.043364 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.043877 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:59.043885 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:59.044168 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 14:22:59.044381 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 14:22:59.044726 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":134,"threads_started":1}
I20260620 14:22:59.045732 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.034773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.0.98.60:39989)
0620 14:22:59.035155 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.035163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.035183 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:22:59.035312 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.035316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.035459 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.035707 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.036431 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.037223 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.038058 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.038417 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.038535 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.041616 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.041626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.041641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.041674 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.043285 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.043733 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.043746 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.043747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.043878 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.044172 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.044176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.044178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.044418 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.044631 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.044852 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.044991 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":130,"threads_started":1}
I20260620 14:22:59.047039 13916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:22:59.052670 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } attempt: 3
Jun 20 14:22:59 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:22:59 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:59.057632 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.047645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36601 (local address 127.0.0.1:54618)
0620 14:22:59.047853 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:59.047872 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:59.047958 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:59.048253 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:59.048257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:59.048448 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:59.048593 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.048601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.049584 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.049588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.050339 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.050348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.050932 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.050936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.051044 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.051629 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:59.051647 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:59.053885 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:59.055887 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.055891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.055894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.056093 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.056417 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.056421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.056423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.056529 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.057069 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:59.057072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:59.057179 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:22:59.057300 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 14:22:59.057436 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 14:22:59.057737 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.047862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.0.98.61:36601)
0620 14:22:59.048004 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.048008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.048035 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:22:59.048080 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.048082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.048135 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.048219 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.048726 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.049476 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.050452 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.050825 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.050926 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.054024 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.054028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.054030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.054065 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.055714 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.056214 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.056217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.056219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.056316 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.056642 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.056645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.056647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.056907 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.056967 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.057384 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.057531 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 14:22:59 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:22:59.058862 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.048157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.0.98.62:41843)
0620 14:22:59.048303 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.048309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.048322 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:22:59.048416 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.048421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.048472 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.048579 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.049256 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.049941 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.050845 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.051199 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.051316 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.054702 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.054706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.054709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.054736 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.056681 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.057197 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.057204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.057206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.057313 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.057674 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.057678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.057680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.057958 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.058173 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.058568 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.058693 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:22:59.058971 13916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:22:59.060716 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } attempt: 3
I20260620 14:22:59.063558 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.053183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52394)
0620 14:22:59.053338 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:59.053355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:59.053439 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:59.053710 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:59.053713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:59.053900 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:59.053980 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.053986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.055008 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.055013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.055867 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.055881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.056576 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.056582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.056684 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.057153 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:59.057167 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:59.058638 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:59.061814 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.061817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.061819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.062054 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.062368 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.062371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.062373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.062481 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.062975 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:59.062980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:59.063095 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 14:22:59.063202 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:22:59.063337 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:22:59.064821 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.053260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.0.98.60:39989)
0620 14:22:59.053386 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.053389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.053403 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:22:59.053537 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.053540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.053583 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.053694 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.054107 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.054863 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.056055 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.056444 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.056581 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.058929 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.058933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.058935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.058959 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.061694 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.062164 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.062170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.062175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.062254 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.062622 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.062626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.062628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.062833 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.062926 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.063941 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.064080 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 14:22:59 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:22:59 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:22:59 dist-test-slave-87nc krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:22:59.070502 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.059502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52398)
0620 14:22:59.059713 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:59.059726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:59.059839 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:59.060362 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:59.060368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:59.060525 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:59.060642 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.060648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.062235 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.062237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.063054 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.063063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.064039 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.064042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.064131 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.064604 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:59.064618 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:59.066032 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:59.068015 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.068019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.068021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.068299 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.068625 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.068631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.068634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.068723 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.069304 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:59.069308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:59.069414 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 14:22:59.069537 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:22:59.070346 (+ 809us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":3}
I20260620 14:22:59.070737 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b: Bootstrap starting.
I20260620 14:22:59.071147 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.060045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.98.61:36601)
0620 14:22:59.060152 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.060156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.060209 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:22:59.060382 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.060385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.060431 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.060492 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.061553 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.062039 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.063099 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.063418 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.063497 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.067199 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.067206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.067213 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.067241 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.069091 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.069612 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.069617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.069618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.069708 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.070054 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.070060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.070062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.070307 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.070476 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.070860 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.070979 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:22:59.071451 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.059713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.0.98.60:39989)
0620 14:22:59.060112 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.060122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.060137 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:22:59.060179 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.060184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.060236 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.060331 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.060774 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.062132 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.063162 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.063624 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.063954 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.066158 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.066164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.066168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.066189 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.067902 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.068413 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.068418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.068421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.068524 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.068843 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.068847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.068849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.069146 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.069262 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.069773 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.071054 (+ 1281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1189,"server-negotiator.queue_time_us":333,"thread_start_us":138,"threads_started":1}
I20260620 14:22:59.071890 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.061113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39989 (local address 127.0.0.1:52402)
0620 14:22:59.061245 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:59.061278 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:59.061360 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:59.062674 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:59.062677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:59.062844 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:59.062931 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.062937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.064216 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.064220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.065081 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:59.065095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.065794 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.065798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:59.065882 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.066360 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:59.066377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:59.067930 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:59.070324 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.070328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.070330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.070589 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.070892 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:59.070896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:59.070898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:59.070977 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:59.071402 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:59.071405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:59.071498 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:22:59.071581 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:22:59.071713 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:22:59.072387 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.061339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.0.98.60:39989)
0620 14:22:59.062463 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.062471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.062482 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:22:59.062511 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.062514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.062548 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.063032 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.063162 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.064098 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.065276 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.065654 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.066092 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.068057 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.068063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.068067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.068090 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.070200 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.070689 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.070692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.070695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.070790 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.071088 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.071091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.071093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.071289 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.072048 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.072122 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.072234 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":1067,"thread_start_us":108,"threads_started":1}
I20260620 14:22:59.073666 13765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:59.074460 13765 log.cc:824] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:59.077243 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b: No bootstrap required, opened a new log
I20260620 14:22:59.077348 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:22:59.078025 13840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff: Bootstrap starting.
I20260620 14:22:59.078035 13916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496: Bootstrap starting.
I20260620 14:22:59.080993 13840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:59.080996 13916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:59.081682 13765 raft_consensus.cc:348] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } }
I20260620 14:22:59.081946 13765 raft_consensus.cc:374] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:59.081946 13840 log.cc:824] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:59.082005 13765 raft_consensus.cc:729] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29da3bda5384342aa006a03451fb55b, State: Initialized, Role: FOLLOWER
I20260620 14:22:59.082218 13916 log.cc:824] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:59.082546 13765 consensus_queue.cc:260] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } }
I20260620 14:22:59.084065 13840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff: No bootstrap required, opened a new log
I20260620 14:22:59.084067 13765 sys_catalog.cc:564] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:59.084362 13916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496: No bootstrap required, opened a new log
I20260620 14:22:59.088609 13840 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } }
I20260620 14:22:59.088861 13840 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:59.088932 13840 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd97cea530a743b0a2c700546381c1ff, State: Initialized, Role: FOLLOWER
I20260620 14:22:59.089540 13840 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } }
I20260620 14:22:59.089838 13916 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } }
I20260620 14:22:59.090168 13916 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:59.090303 13916 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ff2dfbc27348de8dae2cd871ef0496, State: Initialized, Role: FOLLOWER
I20260620 14:22:59.090865 13916 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } }
I20260620 14:22:59.091153 13840 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:59.092270 13916 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:59.092859 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.079653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.0.98.62:41843)
0620 14:22:59.079808 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.079831 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.080887 (+ 1056us) server_negotiation.cc:407] Connection header received
0620 14:22:59.081092 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.081095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.081148 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.081203 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.082237 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.083012 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.084159 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.084472 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.084552 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.088324 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.088327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.088331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.088358 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.090171 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.090695 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.090699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.090701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.090801 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.091735 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.091739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.091741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.092037 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.092171 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.092558 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.092693 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:22:59.095367 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.095522 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:22:59.098843 13956 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:22:59.098847 13968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:22:59.098933 13956 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 14:22:59.099067 13968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:22:59.095254 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.099530 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:22:59.099860 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.100108 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:22:59.102257 13772 master_runner.cc:263] No leader master found from master cd97cea530a743b0a2c700546381c1ff
I20260620 14:22:59.102334 13772 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:22:59.103668 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.080335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.0.98.61:36601)
0620 14:22:59.080434 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.080438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.085242 (+ 4804us) server_negotiation.cc:407] Connection header received
0620 14:22:59.086479 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.086483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.086534 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.086585 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.087385 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.088164 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.093685 (+ 5521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.094022 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.094113 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.098838 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.098842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.098844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.098874 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.101302 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.101862 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.101867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.101869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.101964 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.102330 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.102334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.102336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.102578 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.102704 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.103338 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.103491 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 14:22:59.104113 13943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:22:59.104252 13943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:22:59.090617 13697 master_runner.cc:186] Error getting master registration for 127.0.98.61:36601: OK, instance_id { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" instance_seqno: 1781965378805443 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:22:59.106922 13697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:22:59.108243 13848 master_runner.cc:263] No leader master found from master c2ff2dfbc27348de8dae2cd871ef0496
I20260620 14:22:59.108333 13848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:22:59.110488 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:59.093860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.0.98.60:39989)
0620 14:22:59.093989 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:59.093995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:59.094745 (+ 750us) server_negotiation.cc:407] Connection header received
0620 14:22:59.094948 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:59.094952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:59.095008 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:59.095065 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:59.097002 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.097766 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.099225 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:59.099572 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:59.099653 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:59.105740 (+ 6087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:59.105745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:59.105748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:59.105780 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:59.108358 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.108918 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.108922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.108924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.109014 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:59.109359 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:59.109363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:59.109365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:59.109594 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:59.109710 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:59.110136 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 14:22:59.110297 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:22:59.245237 13944 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:59.245460 13944 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } }
I20260620 14:22:59.246552 13944 leader_election.cc:288] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f29da3bda5384342aa006a03451fb55b (127.0.98.62:41843), c2ff2dfbc27348de8dae2cd871ef0496 (127.0.98.60:39989)
I20260620 14:22:59.247381 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd97cea530a743b0a2c700546381c1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29da3bda5384342aa006a03451fb55b" is_pre_election: true
I20260620 14:22:59.247735 13739 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd97cea530a743b0a2c700546381c1ff in term 0.
I20260620 14:22:59.248242 13788 leader_election.cc:302] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd97cea530a743b0a2c700546381c1ff, f29da3bda5384342aa006a03451fb55b; no voters:
I20260620 14:22:59.248553 13944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:22:59.248656 13944 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:22:59.248708 13944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:59.250233 13944 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } }
I20260620 14:22:59.250941 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd97cea530a743b0a2c700546381c1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29da3bda5384342aa006a03451fb55b"
I20260620 14:22:59.251078 13739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:59.252405 13739 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd97cea530a743b0a2c700546381c1ff in term 1.
I20260620 14:22:59.252777 13788 leader_election.cc:302] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd97cea530a743b0a2c700546381c1ff, f29da3bda5384342aa006a03451fb55b; no voters:
I20260620 14:22:59.253453 13944 leader_election.cc:288] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [CANDIDATE]: Term 1 election: Requested vote from peers f29da3bda5384342aa006a03451fb55b (127.0.98.62:41843), c2ff2dfbc27348de8dae2cd871ef0496 (127.0.98.60:39989)
I20260620 14:22:59.253587 13944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:59.253671 13944 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [term 1 LEADER]: Becoming Leader. State: Replica: cd97cea530a743b0a2c700546381c1ff, State: Running, Role: LEADER
I20260620 14:22:59.254009 13944 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } }
I20260620 14:22:59.255002 13889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd97cea530a743b0a2c700546381c1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" is_pre_election: true
I20260620 14:22:59.255349 13889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd97cea530a743b0a2c700546381c1ff in term 0.
I20260620 14:22:59.255853 13889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd97cea530a743b0a2c700546381c1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ff2dfbc27348de8dae2cd871ef0496"
I20260620 14:22:59.255973 13889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:59.257421 13889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd97cea530a743b0a2c700546381c1ff in term 1.
I20260620 14:22:59.259330 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd97cea530a743b0a2c700546381c1ff. Latest consensus state: current_term: 1 leader_uuid: "cd97cea530a743b0a2c700546381c1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.259469 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:59.264034 13974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:59.265156 13974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 14:22:59.267210 13869 tablet.cc:2369] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:22:59.271924 13889 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [term 1 FOLLOWER]: Refusing update from remote peer cd97cea530a743b0a2c700546381c1ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:59.272792 13972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:59.273691 13739 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [term 1 FOLLOWER]: Refusing update from remote peer cd97cea530a743b0a2c700546381c1ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:59.275887 13972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:59.278087 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd97cea530a743b0a2c700546381c1ff. Latest consensus state: current_term: 1 leader_uuid: "cd97cea530a743b0a2c700546381c1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.278218 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:22:59.279902 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd97cea530a743b0a2c700546381c1ff. Latest consensus state: current_term: 1 leader_uuid: "cd97cea530a743b0a2c700546381c1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.280033 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:22:59.281392 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd97cea530a743b0a2c700546381c1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.281533 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:59.285219 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd97cea530a743b0a2c700546381c1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.285353 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:22:59.285893 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd97cea530a743b0a2c700546381c1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.286010 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:59.286121 13975 mvcc.cc:205] Tried to move back new op lower bound from 7298930193489551360 to 7298930193441656832. Current Snapshot: MvccSnapshot[applied={T|T < 7298930193489551360}]
I20260620 14:22:59.286250 13976 mvcc.cc:205] Tried to move back new op lower bound from 7298930193489551360 to 7298930193441656832. Current Snapshot: MvccSnapshot[applied={T|T < 7298930193489551360}]
I20260620 14:22:59.286876 13978 mvcc.cc:205] Tried to move back new op lower bound from 7298930193489551360 to 7298930193441656832. Current Snapshot: MvccSnapshot[applied={T|T < 7298930193489551360}]
I20260620 14:22:59.287036 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd97cea530a743b0a2c700546381c1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29da3bda5384342aa006a03451fb55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41843 } } peers { permanent_uuid: "cd97cea530a743b0a2c700546381c1ff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36601 } } peers { permanent_uuid: "c2ff2dfbc27348de8dae2cd871ef0496" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39989 } } }
I20260620 14:22:59.287155 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:22:59.289275 13974 catalog_manager.cc:1347] Generated new cluster ID: 025512702da84174bcd5642e25e3b326
I20260620 14:22:59.289347 13974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:22:59.320621 13974 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:22:59.322023 13974 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:59.330135 13974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd97cea530a743b0a2c700546381c1ff: Generated new TSK 0
I20260620 14:22:59.330806 13974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:00.100466 13968 catalog_manager.cc:1259] Loaded cluster ID: 025512702da84174bcd5642e25e3b326
I20260620 14:23:00.100600 13968 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496: loading cluster ID for follower catalog manager: success
I20260620 14:23:00.102342 13968 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496: acquiring CA information for follower catalog manager: success
I20260620 14:23:00.103649 13968 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2ff2dfbc27348de8dae2cd871ef0496: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:23:00.105489 13943 catalog_manager.cc:1259] Loaded cluster ID: 025512702da84174bcd5642e25e3b326
I20260620 14:23:00.105557 13943 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b: loading cluster ID for follower catalog manager: success
I20260620 14:23:00.107123 13943 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b: acquiring CA information for follower catalog manager: success
I20260620 14:23:00.108355 13943 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f29da3bda5384342aa006a03451fb55b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:23:00.398370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 13697
I20260620 14:23:00.405120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 13772
I20260620 14:23:00.411057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 13848
2026-06-20T14:23:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3252 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:00 dist-test-slave-87nc krb5kdc[13989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:00 dist-test-slave-87nc krb5kdc[13989](info): set up 2 sockets
Jun 20 14:23:00 dist-test-slave-87nc krb5kdc[13989](info): commencing operation
krb5kdc: starting...
W20260620 14:23:01.448043 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:01 dist-test-slave-87nc krb5kdc[13989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:01.572885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:01.681389 14002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:01.681653 14002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:01.681692 14002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:01.685609 14002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:01.685667 14002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:01.685688 14002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:01.685703 14002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:01.685719 14002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:01.690946 14002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:01.692238 14002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:01.693190 14002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:01.702133 14008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:01.702121 14007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:01.702121 14010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:01.702507 14002 server_base.cc:1034] running on GCE node
I20260620 14:23:01.703081 14002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:01.704241 14002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:01.705407 14002 hybrid_clock.cc:648] HybridClock initialized: now 1781965381705373 us; error 50 us; skew 500 ppm
Jun 20 14:23:01 dist-test-slave-87nc krb5kdc[13989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:01.708364 14002 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:01.709462 14002 webserver.cc:460] Webserver started at http://127.0.98.62:34395/ using document root <none> and password file <none>
I20260620 14:23:01.710263 14002 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:01.710374 14002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:01.710556 14002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:01.712713 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-87nc"
server_key: "3241fcc49167df4e3c2dcd2c2fa5dc37"
server_key_iv: "a0306af0ea2b71e84e710995d20de036"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.713258 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-87nc"
server_key: "3241fcc49167df4e3c2dcd2c2fa5dc37"
server_key_iv: "a0306af0ea2b71e84e710995d20de036"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.717003 14002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:23:01.719424 14017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.720844 14002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:23:01.720985 14002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-87nc"
server_key: "3241fcc49167df4e3c2dcd2c2fa5dc37"
server_key_iv: "a0306af0ea2b71e84e710995d20de036"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.721102 14002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:01.742677 14002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:01.745787 14002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:01.745946 14002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:01.752857 14002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35687
I20260620 14:23:01.752869 14069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35687 every 8 connection(s)
I20260620 14:23:01.753849 14002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:01.756814 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:01.759750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 14002
I20260620 14:23:01.759948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:01.760236 392 external_mini_cluster.cc:1428] Setting key 186bd6eebb4df5641607e706058ff61d
I20260620 14:23:01.763037 14070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9: Bootstrap starting.
I20260620 14:23:01.765703 14070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:01.766500 14070 log.cc:824] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:01.768636 14070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9: No bootstrap required, opened a new log
Jun 20 14:23:01 dist-test-slave-87nc krb5kdc[13989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:01.772782 14070 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } }
I20260620 14:23:01.773084 14070 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:01.773160 14070 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f6c4b4c57843559ec3d3c7c2b893d9, State: Initialized, Role: FOLLOWER
I20260620 14:23:01.773651 14070 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } }
I20260620 14:23:01.773800 14070 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:01.773866 14070 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:01.773972 14070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:01.774895 14070 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } }
I20260620 14:23:01.775283 14070 leader_election.cc:302] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1f6c4b4c57843559ec3d3c7c2b893d9; no voters:
I20260620 14:23:01.775668 14070 leader_election.cc:288] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:01.775733 14075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:01.775985 14075 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [term 1 LEADER]: Becoming Leader. State: Replica: c1f6c4b4c57843559ec3d3c7c2b893d9, State: Running, Role: LEADER
I20260620 14:23:01.776206 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.761673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.0.98.62:35687)
0620 14:23:01.762130 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.762149 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.762190 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:23:01.762972 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.762995 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.763403 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.763758 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:01.764506 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.765974 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.766868 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.767434 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.767650 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.770137 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:01.770144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:01.770162 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:01.770193 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:01.772942 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:01.773500 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:01.773505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:01.773510 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:01.773628 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:01.773967 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:01.773977 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:01.773979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:01.774448 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:01.774671 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.774959 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.775265 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260620 14:23:01.776512 14075 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } }
I20260620 14:23:01.777029 14070 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:01.778228 14076 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1f6c4b4c57843559ec3d3c7c2b893d9. Latest consensus state: current_term: 1 leader_uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } } }
I20260620 14:23:01.778359 14076 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:01.778651 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6c4b4c57843559ec3d3c7c2b893d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } } }
I20260620 14:23:01.778744 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:01.779134 14084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:01.779242 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:01.781977 14084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:01.787230 14084 catalog_manager.cc:1347] Generated new cluster ID: 51c4fb1cf6f047bdb5ef7f5254ede7c2
I20260620 14:23:01.787319 14084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:01.815276 14084 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:01.816320 14084 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:01.822166 14084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1f6c4b4c57843559ec3d3c7c2b893d9: Generated new TSK 0
I20260620 14:23:01.822881 14084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:01.835624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 14002
2026-06-20T14:23:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1422 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:01 dist-test-slave-87nc krb5kdc[14097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:01 dist-test-slave-87nc krb5kdc[14097](info): set up 2 sockets
Jun 20 14:23:01 dist-test-slave-87nc krb5kdc[14097](info): commencing operation
krb5kdc: starting...
W20260620 14:23:02.872179 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:02 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:02.992610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40601
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:03.101413 14110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:03.101727 14110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:03.101809 14110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:03.105839 14110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:03.105944 14110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:03.105983 14110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:03.106021 14110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:03.106053 14110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:03.111295 14110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:03.112388 14110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:03.113430 14110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:03.121918 14118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:03.121929 14116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:03.121987 14110 server_base.cc:1034] running on GCE node
W20260620 14:23:03.121929 14115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:03.122756 14110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:03.123976 14110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:03.125144 14110 hybrid_clock.cc:648] HybridClock initialized: now 1781965383125129 us; error 29 us; skew 500 ppm
Jun 20 14:23:03 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:03.131714 14110 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:03.132705 14110 webserver.cc:460] Webserver started at http://127.0.98.62:33901/ using document root <none> and password file <none>
I20260620 14:23:03.133297 14110 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:03.133390 14110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:03.133625 14110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:03.135692 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "2b33de6e3cbe4bf99328ca8e61362ddf"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "16744b4359d17d030d67ba33e29aed53"
server_key_iv: "3f3c67fb204e7786dc762d4bf4165bfb"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.136332 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b33de6e3cbe4bf99328ca8e61362ddf"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "16744b4359d17d030d67ba33e29aed53"
server_key_iv: "3f3c67fb204e7786dc762d4bf4165bfb"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.140125 14110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:23:03.142650 14125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.144058 14110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:23:03.144205 14110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "2b33de6e3cbe4bf99328ca8e61362ddf"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "16744b4359d17d030d67ba33e29aed53"
server_key_iv: "3f3c67fb204e7786dc762d4bf4165bfb"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.144346 14110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:03.155141 14110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:03.158043 14110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:03.158252 14110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:03.164928 14110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40601
I20260620 14:23:03.164954 14177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40601 every 8 connection(s)
I20260620 14:23:03.165838 14110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:03.168612 14178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:03.168687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 14110
I20260620 14:23:03.168853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:03.169178 392 external_mini_cluster.cc:1428] Setting key 3c5e616973fb5729274d9019c8b0c779
I20260620 14:23:03.176002 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Bootstrap starting.
I20260620 14:23:03.178341 14178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Neither blocks nor log segments found. Creating new log.
Jun 20 14:23:03 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:03.179131 14178 log.cc:824] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:03.181054 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: No bootstrap required, opened a new log
I20260620 14:23:03.184103 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.170611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.0.98.62:40601)
0620 14:23:03.171049 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:03.171069 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:03.171118 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:23:03.172101 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:03.172126 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:03.172571 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:03.172961 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:03.173684 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.175219 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.176175 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.176668 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.176852 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.179399 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:03.179404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:03.179417 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:03.179444 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:03.181305 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.181747 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.181753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.181765 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.181875 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.182168 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.182172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.182173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.182555 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:03.182753 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:03.183046 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:23:03.183322 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":113,"threads_started":1}
I20260620 14:23:03.184753 14178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:03.184995 14178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:03.185083 14178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b33de6e3cbe4bf99328ca8e61362ddf, State: Initialized, Role: FOLLOWER
I20260620 14:23:03.185578 14178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:03.185725 14178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:03.185811 14178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:03.185931 14178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:03.186965 14178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:03.187309 14178 leader_election.cc:302] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b33de6e3cbe4bf99328ca8e61362ddf; no voters:
I20260620 14:23:03.187618 14178 leader_election.cc:288] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:03.187733 14183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:03.188055 14183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 1 LEADER]: Becoming Leader. State: Replica: 2b33de6e3cbe4bf99328ca8e61362ddf, State: Running, Role: LEADER
I20260620 14:23:03.188421 14183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:03.188896 14178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:03.190111 14185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b33de6e3cbe4bf99328ca8e61362ddf. Latest consensus state: current_term: 1 leader_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } } }
I20260620 14:23:03.190104 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } } }
I20260620 14:23:03.190308 14185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:03.190323 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:03.190691 14192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:03.193755 14192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:03.198971 14192 catalog_manager.cc:1347] Generated new cluster ID: 67b3a0ddab0d4b0faedf87f44026ea02
I20260620 14:23:03.199046 14192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:03.212417 14192 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:03.213753 14192 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:03.224331 14192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Generated new TSK 0
I20260620 14:23:03.225164 14192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:23:03.263659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:03.388862 14204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:03.389122 14204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:03.389164 14204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:03.393316 14204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:03.393378 14204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:03.393435 14204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:23:03.398556 14204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:03.399531 14204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:03.400601 14204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:03.410087 14209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:03.410141 14204 server_base.cc:1034] running on GCE node
W20260620 14:23:03.410374 14212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:03.410377 14210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:03.410923 14204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:03.411490 14204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:03.412663 14204 hybrid_clock.cc:648] HybridClock initialized: now 1781965383412638 us; error 36 us; skew 500 ppm
Jun 20 14:23:03 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:03.415364 14204 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:23:03.416326 14204 webserver.cc:460] Webserver started at http://127.0.98.1:41039/ using document root <none> and password file <none>
I20260620 14:23:03.416891 14204 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:03.416985 14204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:03.417188 14204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:03.419185 14204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "758afbb641df4c788857c4d8f345c6d4"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "5555730b768911cfb2202403b863bb8d"
server_key_iv: "9b74683a6817b6cedb346a2a8494675d"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.419754 14204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "758afbb641df4c788857c4d8f345c6d4"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "5555730b768911cfb2202403b863bb8d"
server_key_iv: "9b74683a6817b6cedb346a2a8494675d"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.423473 14204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:23:03.425931 14219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.427191 14204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:03.427338 14204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "758afbb641df4c788857c4d8f345c6d4"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "5555730b768911cfb2202403b863bb8d"
server_key_iv: "9b74683a6817b6cedb346a2a8494675d"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.427451 14204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:03.438952 14204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:03.442063 14204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:03.442281 14204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:03.442922 14204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:03.443845 14204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:03.443910 14204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.443982 14204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:03.444025 14204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.453568 14204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45805
I20260620 14:23:03.453588 14332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45805 every 8 connection(s)
I20260620 14:23:03.454493 14204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:23:03.461225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 14204
I20260620 14:23:03.461337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:23:03.461619 392 external_mini_cluster.cc:1428] Setting key 7f7f59215ca33be5980a0e29924991a7
Jun 20 14:23:03 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:03.468891 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.456388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38941 (local address 127.0.98.62:40601)
0620 14:23:03.456568 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:03.456572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:03.457718 (+ 1146us) server_negotiation.cc:407] Connection header received
0620 14:23:03.458583 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:03.458586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:03.458632 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:03.458709 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:03.459964 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.460498 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.461709 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.462025 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.462124 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.464878 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:03.464882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:03.464884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:03.464906 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:03.466409 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.467079 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.467082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.467085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.467187 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.467602 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.467604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.467605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.467865 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:03.467944 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:03.468658 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 14:23:03.468766 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:23:03.470252 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.456829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:38941)
0620 14:23:03.457574 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:03.457621 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:03.458387 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:03.458868 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:03.458874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:03.459234 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:03.459748 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.459832 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.460611 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.460614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.461586 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.461596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.462204 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.462210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.462404 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.463098 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:03.463118 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:03.464695 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:03.466536 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:03.466545 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:03.466565 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:03.466952 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:03.467295 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:03.467301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:03.467304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:03.467501 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:03.467976 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:03.467986 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:03.468370 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 14:23:03.468616 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 14:23:03.469046 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:23:03.471632 14333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40601
I20260620 14:23:03.471959 14333 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:03.472491 14333 heartbeater.cc:511] Master 127.0.98.62:40601 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:23:03.474196 14142 ts_manager.cc:194] Registered new tserver with Master: 758afbb641df4c788857c4d8f345c6d4 (127.0.98.1:45805)
I20260620 14:23:03.475387 14142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:23:03.491099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:03.603305 14338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:03.603564 14338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:03.603606 14338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:03.607712 14338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:03.607787 14338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:03.607848 14338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:23:03.613129 14338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:03.614230 14338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:03.615285 14338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:03.624967 14346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:03.624979 14344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:03.624985 14343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:03.625082 14338 server_base.cc:1034] running on GCE node
I20260620 14:23:03.625697 14338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:03.626452 14338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:03.627641 14338 hybrid_clock.cc:648] HybridClock initialized: now 1781965383627621 us; error 32 us; skew 500 ppm
Jun 20 14:23:03 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:03.630621 14338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:23:03.631592 14338 webserver.cc:460] Webserver started at http://127.0.98.2:34987/ using document root <none> and password file <none>
I20260620 14:23:03.632264 14338 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:03.632360 14338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:03.632588 14338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:03.634657 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "db35da665df64cf28fd01a2aadeb90e8"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "c64a992725791071fc7d1f86181c2d09"
server_key_iv: "a67b146bd78fd620b92c18d49124ea46"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.635243 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db35da665df64cf28fd01a2aadeb90e8"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "c64a992725791071fc7d1f86181c2d09"
server_key_iv: "a67b146bd78fd620b92c18d49124ea46"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.638988 14338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:23:03.641408 14353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.642741 14338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:23:03.642884 14338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "db35da665df64cf28fd01a2aadeb90e8"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "c64a992725791071fc7d1f86181c2d09"
server_key_iv: "a67b146bd78fd620b92c18d49124ea46"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.642998 14338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:03.668792 14338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:03.671840 14338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:03.672047 14338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:03.672655 14338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:03.673497 14338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:03.673570 14338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.673638 14338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:03.673682 14338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.682509 14338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35097
I20260620 14:23:03.683521 14466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35097 every 8 connection(s)
I20260620 14:23:03.683800 14338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:23:03.688024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 14338
I20260620 14:23:03.688160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:23:03.688431 392 external_mini_cluster.cc:1428] Setting key ec60b30d0f533a5bd65735ac32360723
Jun 20 14:23:03 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:23:03.698843 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.685626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57515 (local address 127.0.98.62:40601)
0620 14:23:03.685762 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:03.685765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:03.686821 (+ 1056us) server_negotiation.cc:407] Connection header received
0620 14:23:03.687748 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:03.687752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:03.687827 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:03.687906 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:03.689316 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.690093 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.691761 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.692165 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.692283 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.695068 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:03.695071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:03.695073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:03.695096 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:03.696631 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.697229 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.697232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.697233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.697305 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.697714 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.697717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.697718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.697886 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:03.697952 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:03.698616 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 14:23:03.698717 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:23:03.699831 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.685999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:57515)
0620 14:23:03.686676 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:03.686716 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:03.687540 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:03.688072 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:03.688078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:03.688528 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:03.689097 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.689122 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.690214 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.690234 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.691624 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.691635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.692308 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.692311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.692453 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.693029 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:03.693045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:03.694872 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:03.696751 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:03.696759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:03.696777 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:03.697119 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:03.697410 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:03.697413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:03.697415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:03.697617 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:03.698002 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:03.698009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:03.698364 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 14:23:03.698565 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 14:23:03.698832 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:23:03.701026 14467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40601
I20260620 14:23:03.701287 14467 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:03.701779 14467 heartbeater.cc:511] Master 127.0.98.62:40601 requested a full tablet report, sending...
I20260620 14:23:03.702895 14142 ts_manager.cc:194] Registered new tserver with Master: db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097)
I20260620 14:23:03.703477 14142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:23:03.715888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:03.824980 14472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:03.825222 14472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:03.825264 14472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:03.829327 14472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:03.829383 14472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:03.829440 14472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:23:03.834947 14472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:03.836028 14472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:03.837100 14472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:03.846549 14477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:03.846549 14480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:03.846593 14478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:03.846776 14472 server_base.cc:1034] running on GCE node
I20260620 14:23:03.847265 14472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:03.847962 14472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:03.849123 14472 hybrid_clock.cc:648] HybridClock initialized: now 1781965383849108 us; error 30 us; skew 500 ppm
Jun 20 14:23:03 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:03.851841 14472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:23:03.852648 14472 webserver.cc:460] Webserver started at http://127.0.98.3:41453/ using document root <none> and password file <none>
I20260620 14:23:03.853163 14472 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:03.853225 14472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:03.853389 14472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:03.855394 14472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "aa7ea15fb6f143d5889105b47f8ac0fe"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "49b9f84c94f3367f3bd30dc41183bc42"
server_key_iv: "ae8f4c0eeb24e14a60fe87a05b6363d1"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.855993 14472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aa7ea15fb6f143d5889105b47f8ac0fe"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "49b9f84c94f3367f3bd30dc41183bc42"
server_key_iv: "ae8f4c0eeb24e14a60fe87a05b6363d1"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.859586 14472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:23:03.862171 14487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.863438 14472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:03.863543 14472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "aa7ea15fb6f143d5889105b47f8ac0fe"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "49b9f84c94f3367f3bd30dc41183bc42"
server_key_iv: "ae8f4c0eeb24e14a60fe87a05b6363d1"
server_key_version: "encryptionkey@0"
I20260620 14:23:03.863680 14472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:03.889361 14472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:03.892596 14472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:03.892804 14472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:03.893406 14472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:03.894235 14472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:03.894284 14472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.894323 14472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:03.894378 14472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:03.904042 14472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34087
I20260620 14:23:03.904074 14600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34087 every 8 connection(s)
I20260620 14:23:03.904927 14472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:23:03.912328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 14472
I20260620 14:23:03.912460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:23:03.912758 392 external_mini_cluster.cc:1428] Setting key 6393d266bed91c5511f927ee3ba99668
Jun 20 14:23:03 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:03.918540 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.906780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48501 (local address 127.0.98.62:40601)
0620 14:23:03.906925 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:03.906930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:03.908085 (+ 1155us) server_negotiation.cc:407] Connection header received
0620 14:23:03.908947 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:03.908950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:03.909009 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:03.909090 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:03.910197 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.910751 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.912027 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.912380 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.912504 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.915257 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:03.915260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:03.915262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:03.915305 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:03.916754 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.917241 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.917244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.917245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.917319 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.917623 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.917625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.917626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.917795 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:03.917868 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:03.918324 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 14:23:03.918422 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:23:03.919417 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.907221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:48501)
0620 14:23:03.907947 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:03.907990 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:03.908736 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:03.909259 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:03.909265 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:03.909609 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:03.910069 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.910082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.910903 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.910908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.911887 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.911897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.912565 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.912569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.912714 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.913371 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:03.913388 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:03.915110 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:03.916858 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:03.916863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:03.916875 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:03.917148 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:03.917401 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:03.917403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:03.917404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:03.917542 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:03.917879 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:03.917884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:03.918123 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 14:23:03.918279 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 14:23:03.918538 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":134,"threads_started":1}
I20260620 14:23:03.920644 14601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40601
I20260620 14:23:03.920866 14601 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:03.921336 14601 heartbeater.cc:511] Master 127.0.98.62:40601 requested a full tablet report, sending...
I20260620 14:23:03.922304 14142 ts_manager.cc:194] Registered new tserver with Master: aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087)
I20260620 14:23:03.922868 14142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48501
I20260620 14:23:03.926792 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:23:03.936254 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.928852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.0.98.62:40601)
0620 14:23:03.928991 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:03.928996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:03.929168 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:23:03.929282 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:03.929285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:03.929321 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:03.929386 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:03.930099 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.930653 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.931528 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.931738 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.931870 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.933240 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:03.933243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:03.933245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:03.933264 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:03.934734 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.935110 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.935112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.935113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.935185 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:03.935401 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:03.935403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:03.935404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:03.935593 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:03.935688 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:03.935964 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 14:23:03.936075 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:23:03.938999 14142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:23:03.942097 14142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:23:03.957125 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.951486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.0.98.3:34087)
0620 14:23:03.951859 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:03.951871 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:03.952504 (+ 633us) server_negotiation.cc:407] Connection header received
0620 14:23:03.952591 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:03.952596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:03.952765 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:03.952907 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:03.953181 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.953959 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.955825 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.956411 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.956545 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.956695 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:03.956874 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:23:03.956962 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":128,"threads_started":1}
I20260620 14:23:03.957115 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.950913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34087 (local address 127.0.0.1:43480)
0620 14:23:03.952391 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:03.952403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:03.952487 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:03.952954 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:03.952959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:03.952974 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:03.953073 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.953078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.954090 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.954093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.955695 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.955702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.956543 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.956547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.956656 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.956691 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:23:03.956842 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:23:03.956962 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"mutex_wait_us":57,"thread_start_us":77,"threads_started":1}
I20260620 14:23:03.957316 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.950785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.0.98.2:35097)
0620 14:23:03.951600 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:03.951613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:03.951639 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:23:03.951954 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:03.951961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:03.952200 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:03.952311 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:03.952929 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.954160 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.955549 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.956450 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.956555 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.956695 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:03.957097 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 14:23:03.957192 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":94,"threads_started":1}
I20260620 14:23:03.957434 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.951904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.0.98.1:45805)
0620 14:23:03.952243 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:03.952257 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:03.952279 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:23:03.952358 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:03.952363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:03.952556 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:03.952725 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:03.952977 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.954158 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.955664 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.956465 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.956555 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.956695 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:03.957221 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 14:23:03.957298 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":120,"threads_started":1}
I20260620 14:23:03.957595 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.950864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45805 (local address 127.0.0.1:37454)
0620 14:23:03.952097 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:03.952113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:03.952204 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:03.952703 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:03.952706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:03.952723 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:03.952849 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.952858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.954292 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.954295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.955529 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.955541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.956759 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.956763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.956907 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.956948 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:23:03.957031 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:23:03.957098 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"mutex_wait_us":60,"thread_start_us":106,"threads_started":1}
I20260620 14:23:03.957677 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:03.950601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35097 (local address 127.0.0.1:59670)
0620 14:23:03.951510 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:03.951566 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:03.951702 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:03.952598 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:03.952603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:03.952626 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:03.952769 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.952778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.954335 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.954338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.955420 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:03.955428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:03.956759 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:03.956763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:03.956935 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:03.956950 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:23:03.957453 (+ 503us) client_negotiation.cc:240] Negotiation successful
0620 14:23:03.957524 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":49,"thread_start_us":120,"threads_started":1}
I20260620 14:23:03.960745 14267 tablet_service.cc:1467] Processing CreateTablet for tablet d68df7ed3d8b46518fe0f67244ab6c7d (DEFAULT_TABLE table=test-table [id=8d1cc2b18e69431c98b9fa8be20326ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:03.961236 14401 tablet_service.cc:1467] Processing CreateTablet for tablet d68df7ed3d8b46518fe0f67244ab6c7d (DEFAULT_TABLE table=test-table [id=8d1cc2b18e69431c98b9fa8be20326ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:03.961369 14535 tablet_service.cc:1467] Processing CreateTablet for tablet d68df7ed3d8b46518fe0f67244ab6c7d (DEFAULT_TABLE table=test-table [id=8d1cc2b18e69431c98b9fa8be20326ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:03.961925 14267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68df7ed3d8b46518fe0f67244ab6c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:03.962239 14401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68df7ed3d8b46518fe0f67244ab6c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:03.962378 14535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68df7ed3d8b46518fe0f67244ab6c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:03.968293 14618 tablet_bootstrap.cc:492] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: Bootstrap starting.
I20260620 14:23:03.969455 14619 tablet_bootstrap.cc:492] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: Bootstrap starting.
I20260620 14:23:03.970764 14618 tablet_bootstrap.cc:654] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:03.971608 14619 tablet_bootstrap.cc:654] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:03.972085 14620 tablet_bootstrap.cc:492] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Bootstrap starting.
I20260620 14:23:03.972131 14618 log.cc:824] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:03.972491 14619 log.cc:824] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:03.974193 14619 tablet_bootstrap.cc:492] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: No bootstrap required, opened a new log
I20260620 14:23:03.974308 14618 tablet_bootstrap.cc:492] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: No bootstrap required, opened a new log
I20260620 14:23:03.974303 14620 tablet_bootstrap.cc:654] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:03.974406 14619 ts_tablet_manager.cc:1397] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:23:03.974494 14618 ts_tablet_manager.cc:1397] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:23:03.975119 14620 log.cc:824] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:03.977070 14620 tablet_bootstrap.cc:492] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: No bootstrap required, opened a new log
I20260620 14:23:03.977265 14620 ts_tablet_manager.cc:1397] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:23:03.977597 14619 raft_consensus.cc:348] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:03.977847 14619 raft_consensus.cc:374] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:03.977924 14619 raft_consensus.cc:729] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db35da665df64cf28fd01a2aadeb90e8, State: Initialized, Role: FOLLOWER
I20260620 14:23:03.978399 14619 consensus_queue.cc:260] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:03.978292 14618 raft_consensus.cc:348] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:03.978631 14618 raft_consensus.cc:374] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:03.978752 14618 raft_consensus.cc:729] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7ea15fb6f143d5889105b47f8ac0fe, State: Initialized, Role: FOLLOWER
I20260620 14:23:03.979319 14618 consensus_queue.cc:260] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:03.979418 14619 ts_tablet_manager.cc:1428] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:23:03.979835 14467 heartbeater.cc:503] Master 127.0.98.62:40601 was elected leader, sending a full tablet report...
I20260620 14:23:03.980011 14601 heartbeater.cc:503] Master 127.0.98.62:40601 was elected leader, sending a full tablet report...
I20260620 14:23:03.980397 14618 ts_tablet_manager.cc:1428] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:23:03.981739 14620 raft_consensus.cc:348] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:03.981999 14620 raft_consensus.cc:374] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:03.982082 14620 raft_consensus.cc:729] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758afbb641df4c788857c4d8f345c6d4, State: Initialized, Role: FOLLOWER
I20260620 14:23:03.982539 14620 consensus_queue.cc:260] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:03.983398 14333 heartbeater.cc:503] Master 127.0.98.62:40601 was elected leader, sending a full tablet report...
I20260620 14:23:03.983572 14620 ts_tablet_manager.cc:1428] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260620 14:23:04.155918 14602 tablet.cc:2369] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:23:04.184949 14468 tablet.cc:2369] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:23:04.205883 14334 tablet.cc:2369] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:23:04.278857 14626 raft_consensus.cc:482] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:04.279120 14626 raft_consensus.cc:504] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:04.280450 14626 leader_election.cc:288] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097), aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087)
I20260620 14:23:04.284744 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.280829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55751 (local address 127.0.98.2:35097)
0620 14:23:04.280996 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:04.281000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:04.281405 (+ 405us) server_negotiation.cc:407] Connection header received
0620 14:23:04.281520 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:04.281523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:04.281597 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:04.281677 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:04.281958 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.282447 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.283668 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.284101 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.284182 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.284238 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:04.284530 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 14:23:04.284610 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:23:04.284767 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.280760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34087 (local address 127.0.98.1:55141)
0620 14:23:04.281180 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:04.281197 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:04.281315 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:04.281591 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:04.281593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:04.281610 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:04.281754 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.281762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.282579 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.282583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.283542 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.283551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.284337 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.284339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.284439 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.284456 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:23:04.284559 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:23:04.284623 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":43,"thread_start_us":148,"threads_started":1}
I20260620 14:23:04.284775 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.280876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55141 (local address 127.0.98.3:34087)
0620 14:23:04.280994 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:04.281000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:04.281300 (+ 300us) server_negotiation.cc:407] Connection header received
0620 14:23:04.281421 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:04.281424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:04.281483 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:04.281558 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:04.281871 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.282429 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.283677 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.284198 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.284317 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.284384 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:04.284592 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 14:23:04.284669 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:23:04.284770 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.280764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35097 (local address 127.0.98.1:55751)
0620 14:23:04.281289 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:04.281303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:04.281418 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:04.281706 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:04.281709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:04.281729 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:04.281846 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.281854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.282585 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.282587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.283541 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.283550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.284236 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.284242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.284340 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.284414 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:23:04.284486 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:23:04.284574 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":259,"thread_start_us":123,"threads_started":1}
I20260620 14:23:04.285564 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68df7ed3d8b46518fe0f67244ab6c7d" candidate_uuid: "758afbb641df4c788857c4d8f345c6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" is_pre_election: true
I20260620 14:23:04.285564 14421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68df7ed3d8b46518fe0f67244ab6c7d" candidate_uuid: "758afbb641df4c788857c4d8f345c6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db35da665df64cf28fd01a2aadeb90e8" is_pre_election: true
I20260620 14:23:04.285899 14555 raft_consensus.cc:2454] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758afbb641df4c788857c4d8f345c6d4 in term 0.
I20260620 14:23:04.285899 14421 raft_consensus.cc:2454] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758afbb641df4c788857c4d8f345c6d4 in term 0.
I20260620 14:23:04.286439 14222 leader_election.cc:302] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 758afbb641df4c788857c4d8f345c6d4, aa7ea15fb6f143d5889105b47f8ac0fe, db35da665df64cf28fd01a2aadeb90e8; no voters:
I20260620 14:23:04.286717 14626 raft_consensus.cc:2781] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:23:04.286794 14626 raft_consensus.cc:482] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:23:04.286825 14626 raft_consensus.cc:3037] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:04.287968 14626 raft_consensus.cc:504] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:04.288337 14626 leader_election.cc:288] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 1 election: Requested vote from peers db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097), aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087)
I20260620 14:23:04.288691 14421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68df7ed3d8b46518fe0f67244ab6c7d" candidate_uuid: "758afbb641df4c788857c4d8f345c6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db35da665df64cf28fd01a2aadeb90e8"
I20260620 14:23:04.288692 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68df7ed3d8b46518fe0f67244ab6c7d" candidate_uuid: "758afbb641df4c788857c4d8f345c6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe"
I20260620 14:23:04.288839 14555 raft_consensus.cc:3037] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:04.288844 14421 raft_consensus.cc:3037] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:04.290122 14421 raft_consensus.cc:2454] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758afbb641df4c788857c4d8f345c6d4 in term 1.
I20260620 14:23:04.290122 14555 raft_consensus.cc:2454] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758afbb641df4c788857c4d8f345c6d4 in term 1.
I20260620 14:23:04.290437 14222 leader_election.cc:302] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758afbb641df4c788857c4d8f345c6d4, db35da665df64cf28fd01a2aadeb90e8; no voters:
I20260620 14:23:04.290649 14626 raft_consensus.cc:2781] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:04.290875 14626 raft_consensus.cc:686] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 758afbb641df4c788857c4d8f345c6d4, State: Running, Role: LEADER
I20260620 14:23:04.291231 14626 consensus_queue.cc:237] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:04.294370 14140 catalog_manager.cc:5495] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 758afbb641df4c788857c4d8f345c6d4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "758afbb641df4c788857c4d8f345c6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } health_report { overall_health: UNKNOWN } } }
I20260620 14:23:04.319075 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.315580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.98.1:45805)
0620 14:23:04.315705 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:04.315709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:04.315739 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:23:04.315954 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:04.315958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:04.316026 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:04.316131 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:04.316363 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.316885 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.317679 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.317887 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.317963 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.318219 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:04.318621 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:04.318708 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:04.318759 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:23:04.318806 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:23:04.324946 14555 raft_consensus.cc:1261] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [term 1 FOLLOWER]: Refusing update from remote peer 758afbb641df4c788857c4d8f345c6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:04.324999 14421 raft_consensus.cc:1261] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [term 1 FOLLOWER]: Refusing update from remote peer 758afbb641df4c788857c4d8f345c6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:04.325716 14629 consensus_queue.cc:1035] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:04.325902 14626 consensus_queue.cc:1035] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:04.333758 14633 mvcc.cc:205] Tried to move back new op lower bound from 7298930214187335680 to 7298930214059405312. Current Snapshot: MvccSnapshot[applied={T|T < 7298930214187335680}]
I20260620 14:23:04.335037 14634 mvcc.cc:205] Tried to move back new op lower bound from 7298930214187335680 to 7298930214059405312. Current Snapshot: MvccSnapshot[applied={T|T < 7298930214187335680}]
I20260620 14:23:04.341162 14140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:04.341305 14140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:04.343745 14140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Sending DeleteTablet for 3 replicas of tablet d68df7ed3d8b46518fe0f67244ab6c7d
I20260620 14:23:04.344519 14267 tablet_service.cc:1514] Processing DeleteTablet for tablet d68df7ed3d8b46518fe0f67244ab6c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37454
I20260620 14:23:04.344539 14401 tablet_service.cc:1514] Processing DeleteTablet for tablet d68df7ed3d8b46518fe0f67244ab6c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59670
I20260620 14:23:04.344522 14535 tablet_service.cc:1514] Processing DeleteTablet for tablet d68df7ed3d8b46518fe0f67244ab6c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43480
I20260620 14:23:04.345060 14643 tablet_replica.cc:330] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: stopping tablet replica
I20260620 14:23:04.345216 14642 tablet_replica.cc:330] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: stopping tablet replica
I20260620 14:23:04.345314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 14204
I20260620 14:23:04.346665 14643 raft_consensus.cc:2229] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:04.346993 14643 raft_consensus.cc:2258] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:04.348076 14642 raft_consensus.cc:2229] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:04.348387 14642 raft_consensus.cc:2258] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:04.349089 14643 ts_tablet_manager.cc:1905] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:04.349684 14642 ts_tablet_manager.cc:1905] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:04.351997 14643 ts_tablet_manager.cc:1918] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:04.352097 14643 log.cc:1196] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/d68df7ed3d8b46518fe0f67244ab6c7d
W20260620 14:23:04.352334 14126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45805 (error 108)
I20260620 14:23:04.352424 14643 ts_tablet_manager.cc:1939] T d68df7ed3d8b46518fe0f67244ab6c7d P db35da665df64cf28fd01a2aadeb90e8: Deleting consensus metadata
I20260620 14:23:04.352536 14642 ts_tablet_manager.cc:1918] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:04.352634 14642 log.cc:1196] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/d68df7ed3d8b46518fe0f67244ab6c7d
I20260620 14:23:04.352887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 14338
I20260620 14:23:04.352939 14642 ts_tablet_manager.cc:1939] T d68df7ed3d8b46518fe0f67244ab6c7d P aa7ea15fb6f143d5889105b47f8ac0fe: Deleting consensus metadata
I20260620 14:23:04.354010 14127 catalog_manager.cc:4841] TS aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087): tablet d68df7ed3d8b46518fe0f67244ab6c7d (table test-table [id=8d1cc2b18e69431c98b9fa8be20326ab]) successfully deleted
I20260620 14:23:04.356333 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.355986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45805 (local address 127.0.0.1:37460)
0620 14:23:04.356145 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:04.356229 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:23:04.356505 14126 catalog_manager.cc:4584] TS 758afbb641df4c788857c4d8f345c6d4 (127.0.98.1:45805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d68df7ed3d8b46518fe0f67244ab6c7d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45805: connect: Connection refused (error 111)
I20260620 14:23:04.359614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 14472
I20260620 14:23:04.359915 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.359631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35097 (local address 127.0.0.1:59678)
0620 14:23:04.359750 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:04.359823 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:23:04.365358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 14110
I20260620 14:23:04.372181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40601
--webserver_interface=127.0.98.62
--webserver_port=33901
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:04.484081 14645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:04.484404 14645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:04.484488 14645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:04.488587 14645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:04.488668 14645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:04.488705 14645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:04.488770 14645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:04.488821 14645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:04.495507 14645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:04.496662 14645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:04.497735 14645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:04.510876 14654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:04.510876 14652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:04.510903 14651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:04.511010 14645 server_base.cc:1034] running on GCE node
I20260620 14:23:04.511942 14645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:04.513079 14645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:04.514240 14645 hybrid_clock.cc:648] HybridClock initialized: now 1781965384514221 us; error 31 us; skew 500 ppm
Jun 20 14:23:04 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:04.517081 14645 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:04.517975 14645 webserver.cc:460] Webserver started at http://127.0.98.62:33901/ using document root <none> and password file <none>
I20260620 14:23:04.518581 14645 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:04.518682 14645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:04.522588 14645 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:04.524778 14661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:04.526142 14645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:23:04.526299 14645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "2b33de6e3cbe4bf99328ca8e61362ddf"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "16744b4359d17d030d67ba33e29aed53"
server_key_iv: "3f3c67fb204e7786dc762d4bf4165bfb"
server_key_version: "encryptionkey@0"
I20260620 14:23:04.526980 14645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:04.545933 14645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:04.548789 14645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:04.548997 14645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:04.556478 14645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40601
I20260620 14:23:04.556789 14713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40601 every 8 connection(s)
I20260620 14:23:04.557690 14645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:04.558621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 14645
I20260620 14:23:04.559127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45805
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40601
--webserver_port=41039
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:04.563092 14714 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:23:04.564815 14714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Bootstrap starting.
I20260620 14:23:04.581656 14714 log.cc:824] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:04.590445 14714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:23:04.590873 14714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Bootstrap complete.
I20260620 14:23:04.595067 14714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:04.595506 14714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b33de6e3cbe4bf99328ca8e61362ddf, State: Initialized, Role: FOLLOWER
I20260620 14:23:04.596045 14714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:04.596194 14714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:04.596293 14714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:04.596411 14714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:23:04.598343 14714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:04.598690 14714 leader_election.cc:302] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b33de6e3cbe4bf99328ca8e61362ddf; no voters:
I20260620 14:23:04.599028 14714 leader_election.cc:288] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:23:04.599141 14718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:23:04.599347 14718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 2 LEADER]: Becoming Leader. State: Replica: 2b33de6e3cbe4bf99328ca8e61362ddf, State: Running, Role: LEADER
I20260620 14:23:04.599637 14718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:04.600299 14714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:04.600661 14718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b33de6e3cbe4bf99328ca8e61362ddf. Latest consensus state: current_term: 2 leader_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } } }
I20260620 14:23:04.600778 14718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:04.602170 14722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:04.602684 14719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } } }
I20260620 14:23:04.602777 14719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:04.606853 14722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:04.607478 14722 catalog_manager.cc:1259] Loaded cluster ID: 67b3a0ddab0d4b0faedf87f44026ea02
I20260620 14:23:04.607543 14722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:04.608983 14722 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:23:04.613821 14722 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:23:04.619269 14722 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:04.620038 14722 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Loaded TSK: 0
I20260620 14:23:04.621218 14722 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:23:04.677086 14716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:04.677399 14716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:04.677477 14716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:04.681631 14716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:04.681725 14716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:04.681819 14716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:23:04.687366 14716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41039
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:04.688578 14716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:04.689637 14716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:04.700227 14741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:04.700290 14744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:04.700227 14742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:04.700587 14716 server_base.cc:1034] running on GCE node
I20260620 14:23:04.701231 14716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:04.702073 14716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:04.703336 14716 hybrid_clock.cc:648] HybridClock initialized: now 1781965384703307 us; error 64 us; skew 500 ppm
Jun 20 14:23:04 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:04.706846 14716 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:23:04.707827 14716 webserver.cc:460] Webserver started at http://127.0.98.1:41039/ using document root <none> and password file <none>
I20260620 14:23:04.708369 14716 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:04.708457 14716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:04.712502 14716 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:23:04.714671 14751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:04.716080 14716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:23:04.716209 14716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "758afbb641df4c788857c4d8f345c6d4"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "5555730b768911cfb2202403b863bb8d"
server_key_iv: "9b74683a6817b6cedb346a2a8494675d"
server_key_version: "encryptionkey@0"
I20260620 14:23:04.716782 14716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:04.729029 14716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:04.732510 14716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:04.732705 14716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:04.733271 14716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:04.734351 14758 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:23:04.739313 14716 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:23:04.739414 14716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 14:23:04.739535 14716 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:23:04.741386 14716 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:23:04.741459 14716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 14:23:04.741750 14758 tablet_bootstrap.cc:492] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Bootstrap starting.
I20260620 14:23:04.753043 14716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45805
I20260620 14:23:04.753307 14865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45805 every 8 connection(s)
I20260620 14:23:04.754169 14716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:23:04.756953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 14716
I20260620 14:23:04.757477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35097
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40601
--webserver_port=34987
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:04.763856 14758 log.cc:824] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:23:04 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:04.772220 14758 tablet_bootstrap.cc:492] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:23:04.772691 14758 tablet_bootstrap.cc:492] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Bootstrap complete.
I20260620 14:23:04.773869 14758 ts_tablet_manager.cc:1397] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.012s
I20260620 14:23:04.776695 14758 raft_consensus.cc:348] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:04.777240 14758 raft_consensus.cc:729] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758afbb641df4c788857c4d8f345c6d4, State: Initialized, Role: FOLLOWER
I20260620 14:23:04.777688 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.756398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:38505)
0620 14:23:04.758283 (+ 1885us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:04.758345 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:04.759702 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:04.761131 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:04.761139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:04.761720 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:04.762435 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.762459 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.764825 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.764833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.766580 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.766596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.768124 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.768131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.768384 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.769126 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:04.769154 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:04.771275 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:04.774035 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:04.774041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:04.774055 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:04.774416 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:04.774849 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:04.774852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:04.774855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:04.775030 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:04.775738 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:04.775745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:04.776084 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 14:23:04.776297 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 14:23:04.776672 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1642,"thread_start_us":143,"threads_started":1}
I20260620 14:23:04.777820 14758 consensus_queue.cc:260] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } }
I20260620 14:23:04.778353 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.756344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38505 (local address 127.0.98.62:40601)
0620 14:23:04.757922 (+ 1578us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:04.757948 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:04.758468 (+ 520us) server_negotiation.cc:407] Connection header received
0620 14:23:04.760266 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:04.760285 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:04.760725 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:04.761095 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:04.762664 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.764593 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.766753 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.767651 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.767875 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.771498 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:04.771517 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:04.771539 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:04.771584 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:04.773881 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:04.774554 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:04.774560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:04.774565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:04.774729 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:04.775141 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:04.775144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:04.775146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:04.775611 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:04.775755 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:04.776473 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 14:23:04.776781 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":135,"threads_started":1}
I20260620 14:23:04.778859 14758 ts_tablet_manager.cc:1428] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:23:04.780763 14866 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40601
I20260620 14:23:04.781005 14866 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:04.781554 14866 heartbeater.cc:511] Master 127.0.98.62:40601 requested a full tablet report, sending...
I20260620 14:23:04.784124 14678 ts_manager.cc:194] Registered new tserver with Master: 758afbb641df4c788857c4d8f345c6d4 (127.0.98.1:45805)
I20260620 14:23:04.787472 14678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38505
I20260620 14:23:04.789927 14866 heartbeater.cc:503] Master 127.0.98.62:40601 was elected leader, sending a full tablet report...
Jun 20 14:23:04 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:23:04.797912 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.787004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.0.98.1:45805)
0620 14:23:04.787387 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:04.787398 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:04.787843 (+ 445us) server_negotiation.cc:407] Connection header received
0620 14:23:04.788074 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:04.788080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:04.788300 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:04.788450 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:04.789168 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.790362 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.791279 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.791800 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.791905 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.794223 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:04.794232 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:04.794242 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:04.794264 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:04.795946 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:04.796443 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:04.796449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:04.796451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:04.796538 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:04.796845 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:04.796849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:04.796850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:04.797098 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:04.797266 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:04.797599 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:23:04.797763 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":123,"threads_started":1}
I20260620 14:23:04.797912 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.786919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45805 (local address 127.0.0.1:37476)
0620 14:23:04.787676 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:04.787728 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:04.787913 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:04.788464 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:04.788470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:04.788912 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:04.789037 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.789046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.790487 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.790491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.791144 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.791154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.791894 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.791910 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.792036 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.792625 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:04.792637 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:04.794060 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:04.796056 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:04.796066 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:04.796072 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:04.796321 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:04.796644 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:04.796649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:04.796651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:04.796745 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:04.797214 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:04.797221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:04.797362 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:23:04.797505 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 14:23:04.797725 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":126,"threads_started":1}
I20260620 14:23:04.798942 14800 tablet_service.cc:1514] Processing DeleteTablet for tablet d68df7ed3d8b46518fe0f67244ab6c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37476
I20260620 14:23:04.800173 14875 tablet_replica.cc:330] stopping tablet replica
I20260620 14:23:04.800498 14875 raft_consensus.cc:2229] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:04.800769 14875 raft_consensus.cc:2258] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:04.801878 14875 ts_tablet_manager.cc:1905] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:04.805558 14875 ts_tablet_manager.cc:1918] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:04.805673 14875 log.cc:1196] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/d68df7ed3d8b46518fe0f67244ab6c7d
I20260620 14:23:04.805967 14875 ts_tablet_manager.cc:1939] T d68df7ed3d8b46518fe0f67244ab6c7d P 758afbb641df4c788857c4d8f345c6d4: Deleting consensus metadata
I20260620 14:23:04.807574 14662 catalog_manager.cc:4841] TS 758afbb641df4c788857c4d8f345c6d4 (127.0.98.1:45805): tablet d68df7ed3d8b46518fe0f67244ab6c7d (table test-table [id=8d1cc2b18e69431c98b9fa8be20326ab]) successfully deleted
W20260620 14:23:04.882506 14870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:04.882773 14870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:04.882817 14870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:04.886819 14870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:04.886883 14870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:04.886943 14870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:23:04.892288 14870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34987
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:04.893395 14870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:04.894476 14870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:04.904497 14883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:04.904486 14881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:04.904528 14880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:04.904909 14870 server_base.cc:1034] running on GCE node
I20260620 14:23:04.905313 14870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:04.905957 14870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:04.907147 14870 hybrid_clock.cc:648] HybridClock initialized: now 1781965384907129 us; error 39 us; skew 500 ppm
Jun 20 14:23:04 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:04.909875 14870 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:23:04.910759 14870 webserver.cc:460] Webserver started at http://127.0.98.2:34987/ using document root <none> and password file <none>
I20260620 14:23:04.911321 14870 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:04.911434 14870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:04.915292 14870 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:04.917471 14890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:04.918772 14870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:04.918932 14870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "db35da665df64cf28fd01a2aadeb90e8"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "c64a992725791071fc7d1f86181c2d09"
server_key_iv: "a67b146bd78fd620b92c18d49124ea46"
server_key_version: "encryptionkey@0"
I20260620 14:23:04.919500 14870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:04.933956 14870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:04.937330 14870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:04.937552 14870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:04.938165 14870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:04.939024 14870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:04.939097 14870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:04.939169 14870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:04.939219 14870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:04.948218 14870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35097
I20260620 14:23:04.948243 15003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35097 every 8 connection(s)
I20260620 14:23:04.949285 14870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:23:04.956336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 14870
I20260620 14:23:04.956773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34087
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40601
--webserver_port=41453
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:23:04 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:04.962989 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.951184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43903 (local address 127.0.98.62:40601)
0620 14:23:04.951326 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:04.951330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:04.952425 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 14:23:04.953358 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:04.953362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:04.953426 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:04.953538 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:04.954748 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.955285 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.956508 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.956755 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.956872 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.959339 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:04.959343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:04.959345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:04.959371 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:04.960921 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:04.961444 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:04.961448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:04.961450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:04.961532 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:04.961879 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:04.961883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:04.961885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:04.962068 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:04.962175 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:04.962672 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 14:23:04.962810 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:23:04.963948 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.951549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:43903)
0620 14:23:04.952259 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:04.952303 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:04.953136 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:04.953733 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:04.953740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:04.954106 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:04.954585 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.954598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.955415 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.955418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.956378 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:04.956387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.956913 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.956917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:04.957090 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.957719 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:04.957736 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:04.959160 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:04.961039 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:04.961045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:04.961061 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:04.961338 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:04.961625 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:04.961629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:04.961631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:04.961782 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:04.962173 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:04.962180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:04.962433 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 14:23:04.962620 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 14:23:04.962964 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":174,"threads_started":1}
I20260620 14:23:04.965194 15004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40601
I20260620 14:23:04.965487 15004 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:04.966164 15004 heartbeater.cc:511] Master 127.0.98.62:40601 requested a full tablet report, sending...
I20260620 14:23:04.967397 14678 ts_manager.cc:194] Registered new tserver with Master: db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097)
I20260620 14:23:04.968050 14678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43903
W20260620 14:23:05.071265 15007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:05.071520 15007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:05.071561 15007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:05.075580 15007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:05.075654 15007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:05.075713 15007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:23:05.081140 15007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41453
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:05.082280 15007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:05.083446 15007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:05.093323 15015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:05.093314 15012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:05.093314 15013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:05.093508 15007 server_base.cc:1034] running on GCE node
I20260620 14:23:05.094132 15007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:05.094801 15007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:05.095996 15007 hybrid_clock.cc:648] HybridClock initialized: now 1781965385095960 us; error 57 us; skew 500 ppm
Jun 20 14:23:05 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:05.098901 15007 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:23:05.099852 15007 webserver.cc:460] Webserver started at http://127.0.98.3:41453/ using document root <none> and password file <none>
I20260620 14:23:05.100425 15007 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:05.100541 15007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:05.104627 15007 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:23:05.106629 15022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:05.107942 15007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:23:05.108071 15007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "aa7ea15fb6f143d5889105b47f8ac0fe"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "49b9f84c94f3367f3bd30dc41183bc42"
server_key_iv: "ae8f4c0eeb24e14a60fe87a05b6363d1"
server_key_version: "encryptionkey@0"
I20260620 14:23:05.108616 15007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:05.127882 15007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:05.131346 15007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:05.131583 15007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:05.132247 15007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:05.133060 15007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:05.133131 15007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:05.133198 15007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:05.133251 15007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:05.142167 15007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34087
I20260620 14:23:05.142189 15135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34087 every 8 connection(s)
I20260620 14:23:05.143162 15007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:23:05.143862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 15007
I20260620 14:23:05.154073 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.144643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.0.98.62:40601)
0620 14:23:05.144795 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.144800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.145145 (+ 345us) server_negotiation.cc:407] Connection header received
0620 14:23:05.145295 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.145298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.145351 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.145433 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:05.146372 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.147143 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.148248 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.148579 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.148692 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.150415 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:05.150418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:05.150421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:05.150454 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:05.152126 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:05.152651 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:05.152655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:05.152657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:05.152751 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:05.153050 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:05.153053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:05.153055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:05.153300 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:05.153515 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.153720 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.153840 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 14:23:05 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:05.158721 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.145069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52337 (local address 127.0.98.62:40601)
0620 14:23:05.145311 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.145315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.146617 (+ 1302us) server_negotiation.cc:407] Connection header received
0620 14:23:05.147660 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.147663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.147712 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.147842 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:05.149270 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.150202 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.152023 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.152375 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.152513 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.155444 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:05.155448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:05.155450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:05.155476 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:05.156889 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:05.157363 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:05.157365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:05.157366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:05.157437 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:05.157765 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:05.157767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:05.157768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:05.157928 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:05.158003 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.158497 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.158594 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260620 14:23:05.159658 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.145642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:52337)
0620 14:23:05.146400 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:05.146456 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:05.147448 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:05.148007 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:05.148014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:05.148449 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:05.149080 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.149100 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.150388 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.150392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.151875 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.151889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.152537 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.152539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.152760 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.153352 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:05.153377 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:05.155253 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:05.157008 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:05.157013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:05.157026 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:05.157275 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:05.157526 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:05.157528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:05.157529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:05.157684 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:05.158041 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:05.158046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:05.158289 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 14:23:05.158453 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 14:23:05.158712 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":156,"threads_started":1}
I20260620 14:23:05.160789 15136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40601
I20260620 14:23:05.161078 15136 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:05.161592 15136 heartbeater.cc:511] Master 127.0.98.62:40601 requested a full tablet report, sending...
I20260620 14:23:05.162652 14678 ts_manager.cc:194] Registered new tserver with Master: aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087)
I20260620 14:23:05.163232 14678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52337
I20260620 14:23:05.168114 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:23:05.172312 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.168702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.0.98.62:40601)
0620 14:23:05.168830 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.168834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.169026 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:23:05.169142 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.169144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.169229 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.169314 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:05.169532 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.170037 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.170869 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.171097 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.171174 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.171487 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:05.171980 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:05.172096 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.172156 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.172209 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:23:05.172967 14678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:23:05.176306 14678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:23:05.184394 14800 tablet_service.cc:1467] Processing CreateTablet for tablet c83a5ad155cf4b9a940295ffbb17b6d0 (DEFAULT_TABLE table=test-table [id=9081cdd415d34a3da5cc38e66f263a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:05.185109 14800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83a5ad155cf4b9a940295ffbb17b6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:05.187652 14758 tablet_bootstrap.cc:492] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Bootstrap starting.
I20260620 14:23:05.188462 14758 tablet_bootstrap.cc:654] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:05.189446 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.183229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35097 (local address 127.0.0.1:59694)
0620 14:23:05.183343 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:05.183359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:05.183447 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:05.184422 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:05.184428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:05.184451 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:05.184562 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.184571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.186282 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.186286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.187756 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.187789 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.188900 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.188906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.189039 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.189090 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:23:05.189179 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:23:05.189274 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":140}
I20260620 14:23:05.189615 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.183547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34087 (local address 127.0.0.1:43488)
0620 14:23:05.183940 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:05.183954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:05.184036 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:05.184826 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:05.184830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:05.184847 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:05.184943 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.184950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.186226 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.186230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.188166 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.188175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.189202 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.189206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.189299 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.189321 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:23:05.189395 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:23:05.189472 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":508,"thread_start_us":100,"threads_started":1}
I20260620 14:23:05.190188 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.183974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.98.3:34087)
0620 14:23:05.184290 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.184303 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.184330 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:23:05.184408 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.184413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.184670 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.184837 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:05.185089 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.186091 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.188292 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.189079 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.189186 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.189321 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.189873 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.190010 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":125,"threads_started":1}
I20260620 14:23:05.190541 14758 tablet_bootstrap.cc:492] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: No bootstrap required, opened a new log
I20260620 14:23:05.190618 14758 ts_tablet_manager.cc:1397] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:23:05.190964 14758 raft_consensus.cc:348] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.190244 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.183449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.0.98.2:35097)
0620 14:23:05.183848 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.183862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.183887 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:23:05.183973 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.183981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.184244 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.184421 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:05.184717 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.186111 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.187922 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.188752 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.188868 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.189037 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.189857 (+ 820us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.190111 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":188,"threads_started":1}
I20260620 14:23:05.191056 14758 raft_consensus.cc:374] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:05.191084 14758 raft_consensus.cc:729] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758afbb641df4c788857c4d8f345c6d4, State: Initialized, Role: FOLLOWER
I20260620 14:23:05.191226 14758 consensus_queue.cc:260] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.191506 14758 ts_tablet_manager.cc:1428] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:05.193413 15070 tablet_service.cc:1467] Processing CreateTablet for tablet c83a5ad155cf4b9a940295ffbb17b6d0 (DEFAULT_TABLE table=test-table [id=9081cdd415d34a3da5cc38e66f263a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:05.193882 14938 tablet_service.cc:1467] Processing CreateTablet for tablet c83a5ad155cf4b9a940295ffbb17b6d0 (DEFAULT_TABLE table=test-table [id=9081cdd415d34a3da5cc38e66f263a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:05.194550 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83a5ad155cf4b9a940295ffbb17b6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:05.194875 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83a5ad155cf4b9a940295ffbb17b6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:05.202073 14872 raft_consensus.cc:482] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:05.202327 14872 raft_consensus.cc:504] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.202905 15149 tablet_bootstrap.cc:492] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: Bootstrap starting.
I20260620 14:23:05.203334 15150 tablet_bootstrap.cc:492] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: Bootstrap starting.
I20260620 14:23:05.203665 14872 leader_election.cc:288] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097), aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087)
I20260620 14:23:05.205948 15149 tablet_bootstrap.cc:654] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:05.206911 15150 tablet_bootstrap.cc:654] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:05.207259 15149 log.cc:824] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:05.207808 15150 log.cc:824] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:05.208389 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.203920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34087 (local address 127.0.98.1:53171)
0620 14:23:05.204031 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:05.204051 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:05.204162 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:05.204488 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:05.204491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:05.204512 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:05.204633 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.204640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.205601 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.205604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.206932 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.206943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.207910 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.207915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.208030 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.208092 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 14:23:05.208144 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:23:05.208232 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:23:05.208880 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.203994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53171 (local address 127.0.98.3:34087)
0620 14:23:05.204202 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.204206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.204250 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:23:05.204296 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.204299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.204352 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.204438 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:05.204770 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.205462 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.207102 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.207727 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.208571 (+ 844us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.208633 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.208667 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.208759 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 14:23:05.210191 15149 tablet_bootstrap.cc:492] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: No bootstrap required, opened a new log
I20260620 14:23:05.210389 15150 tablet_bootstrap.cc:492] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: No bootstrap required, opened a new log
I20260620 14:23:05.210433 15149 ts_tablet_manager.cc:1397] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 14:23:05.209368 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83a5ad155cf4b9a940295ffbb17b6d0" candidate_uuid: "758afbb641df4c788857c4d8f345c6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" is_pre_election: true
I20260620 14:23:05.210687 15150 ts_tablet_manager.cc:1397] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
W20260620 14:23:05.211989 14753 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087): Illegal state: must be running to vote when last-logged opid is not known
I20260620 14:23:05.213444 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.203923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35097 (local address 127.0.98.1:48425)
0620 14:23:05.208011 (+ 4088us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:05.208027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:05.208163 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:05.208540 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:05.208543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:05.208560 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:05.208665 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.208672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.209726 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.209729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.211728 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.211740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.212900 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.212904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.213019 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.213043 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:23:05.213182 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:23:05.213268 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4016,"mutex_wait_us":166,"thread_start_us":122,"threads_started":1}
I20260620 14:23:05.213230 15149 raft_consensus.cc:348] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.213521 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.204092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48425 (local address 127.0.98.2:35097)
0620 14:23:05.207796 (+ 3704us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.207803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.208094 (+ 291us) server_negotiation.cc:407] Connection header received
0620 14:23:05.208293 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.208297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.208344 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.208393 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:05.208761 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.209572 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.211905 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.212635 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.212725 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.212791 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.213207 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.213414 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3588}
I20260620 14:23:05.213533 15149 raft_consensus.cc:374] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:05.213619 15149 raft_consensus.cc:729] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7ea15fb6f143d5889105b47f8ac0fe, State: Initialized, Role: FOLLOWER
I20260620 14:23:05.214115 15149 consensus_queue.cc:260] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.214334 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83a5ad155cf4b9a940295ffbb17b6d0" candidate_uuid: "758afbb641df4c788857c4d8f345c6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db35da665df64cf28fd01a2aadeb90e8" is_pre_election: true
I20260620 14:23:05.214864 15136 heartbeater.cc:503] Master 127.0.98.62:40601 was elected leader, sending a full tablet report...
W20260620 14:23:05.215087 14754 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097): Illegal state: must be running to vote when last-logged opid is not known
I20260620 14:23:05.215186 14754 leader_election.cc:302] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 758afbb641df4c788857c4d8f345c6d4; no voters: aa7ea15fb6f143d5889105b47f8ac0fe, db35da665df64cf28fd01a2aadeb90e8
I20260620 14:23:05.215267 15149 ts_tablet_manager.cc:1428] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 14:23:05.215449 14872 raft_consensus.cc:2726] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:23:05.216454 15150 raft_consensus.cc:348] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.216692 15150 raft_consensus.cc:374] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:05.216742 15150 raft_consensus.cc:729] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db35da665df64cf28fd01a2aadeb90e8, State: Initialized, Role: FOLLOWER
I20260620 14:23:05.217175 15150 consensus_queue.cc:260] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.218146 15004 heartbeater.cc:503] Master 127.0.98.62:40601 was elected leader, sending a full tablet report...
I20260620 14:23:05.219085 15150 ts_tablet_manager.cc:1428] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
W20260620 14:23:05.255628 14867 tablet.cc:2369] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:23:05.394266 15137 tablet.cc:2369] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:23:05.450556 15005 tablet.cc:2369] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:23:05.501791 15154 raft_consensus.cc:482] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:05.502050 15154 raft_consensus.cc:504] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.503154 15154 leader_election.cc:288] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097), 758afbb641df4c788857c4d8f345c6d4 (127.0.98.1:45805)
I20260620 14:23:05.507104 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.503415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45805 (local address 127.0.98.3:36737)
0620 14:23:05.503521 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:05.503537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:05.503632 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:05.504080 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:05.504084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:05.504102 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:05.504212 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.504219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.505024 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.505027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.505884 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.505892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.506737 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.506740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.506817 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.506838 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:23:05.506889 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:23:05.506947 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:23:05.507192 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.503508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36737 (local address 127.0.98.1:45805)
0620 14:23:05.503842 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.503846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.503864 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:23:05.503926 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.503930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.503980 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.504073 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:05.504337 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.504892 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.506019 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.506615 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.506728 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.506889 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.506926 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.507016 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
I20260620 14:23:05.507551 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.503508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43709 (local address 127.0.98.2:35097)
0620 14:23:05.503641 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.503646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.504191 (+ 545us) server_negotiation.cc:407] Connection header received
0620 14:23:05.504345 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.504348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.504413 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.504491 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:05.504771 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.505289 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.506381 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.506945 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.507045 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.507102 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.507340 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.507424 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:23:05.507831 14820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83a5ad155cf4b9a940295ffbb17b6d0" candidate_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758afbb641df4c788857c4d8f345c6d4" is_pre_election: true
I20260620 14:23:05.507956 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.503415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35097 (local address 127.0.98.3:43709)
0620 14:23:05.504086 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:05.504098 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:05.504232 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:05.504501 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:05.504504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:05.504519 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:05.504635 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.504648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.505409 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.505415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.506241 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:05.506253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.507071 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.507074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:05.507177 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.507208 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:23:05.507729 (+ 521us) client_negotiation.cc:240] Negotiation successful
0620 14:23:05.507833 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":318,"thread_start_us":126,"threads_started":1}
I20260620 14:23:05.508121 14820 raft_consensus.cc:2454] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7ea15fb6f143d5889105b47f8ac0fe in term 0.
I20260620 14:23:05.508572 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83a5ad155cf4b9a940295ffbb17b6d0" candidate_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db35da665df64cf28fd01a2aadeb90e8" is_pre_election: true
I20260620 14:23:05.508589 15023 leader_election.cc:302] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758afbb641df4c788857c4d8f345c6d4, aa7ea15fb6f143d5889105b47f8ac0fe; no voters:
I20260620 14:23:05.508872 14958 raft_consensus.cc:2454] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7ea15fb6f143d5889105b47f8ac0fe in term 0.
I20260620 14:23:05.508884 15154 raft_consensus.cc:2781] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:23:05.508977 15154 raft_consensus.cc:482] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:23:05.509039 15154 raft_consensus.cc:3037] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:05.510130 15154 raft_consensus.cc:504] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.510524 15154 leader_election.cc:288] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [CANDIDATE]: Term 1 election: Requested vote from peers db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097), 758afbb641df4c788857c4d8f345c6d4 (127.0.98.1:45805)
I20260620 14:23:05.510838 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83a5ad155cf4b9a940295ffbb17b6d0" candidate_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db35da665df64cf28fd01a2aadeb90e8"
I20260620 14:23:05.510908 14820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83a5ad155cf4b9a940295ffbb17b6d0" candidate_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758afbb641df4c788857c4d8f345c6d4"
I20260620 14:23:05.510955 14958 raft_consensus.cc:3037] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:05.511085 14820 raft_consensus.cc:3037] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:05.512138 14958 raft_consensus.cc:2454] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa7ea15fb6f143d5889105b47f8ac0fe in term 1.
I20260620 14:23:05.512274 14820 raft_consensus.cc:2454] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa7ea15fb6f143d5889105b47f8ac0fe in term 1.
I20260620 14:23:05.512449 15025 leader_election.cc:302] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa7ea15fb6f143d5889105b47f8ac0fe, db35da665df64cf28fd01a2aadeb90e8; no voters:
I20260620 14:23:05.512619 15154 raft_consensus.cc:2781] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:05.512840 15154 raft_consensus.cc:686] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 1 LEADER]: Becoming Leader. State: Replica: aa7ea15fb6f143d5889105b47f8ac0fe, State: Running, Role: LEADER
I20260620 14:23:05.513164 15154 consensus_queue.cc:237] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:05.516309 14677 catalog_manager.cc:5495] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe reported cstate change: term changed from 0 to 1, leader changed from <none> to aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } health_report { overall_health: UNKNOWN } } }
I20260620 14:23:05.558570 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.555038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.98.3:34087)
0620 14:23:05.555190 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.555194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.555207 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:23:05.555281 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.555283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.555336 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.555427 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:05.555631 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.556151 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.557053 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:05.557280 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:05.557394 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:05.557736 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:05.558148 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:05.558292 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.558332 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.558392 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:23:05.564457 14958 raft_consensus.cc:1261] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [term 1 FOLLOWER]: Refusing update from remote peer aa7ea15fb6f143d5889105b47f8ac0fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:05.564530 14820 raft_consensus.cc:1261] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Refusing update from remote peer aa7ea15fb6f143d5889105b47f8ac0fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:05.565155 15154 consensus_queue.cc:1035] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:05.565590 15158 consensus_queue.cc:1035] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:05.574255 15163 mvcc.cc:205] Tried to move back new op lower bound from 7298930219264503808 to 7298930219064549376. Current Snapshot: MvccSnapshot[applied={T|T < 7298930219264503808}]
I20260620 14:23:05.574441 15161 mvcc.cc:205] Tried to move back new op lower bound from 7298930219264503808 to 7298930219064549376. Current Snapshot: MvccSnapshot[applied={T|T < 7298930219264503808}]
I20260620 14:23:05.583004 14677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:05.583158 14677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:05.585646 14677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Sending DeleteTablet for 3 replicas of tablet c83a5ad155cf4b9a940295ffbb17b6d0
I20260620 14:23:05.586386 15070 tablet_service.cc:1514] Processing DeleteTablet for tablet c83a5ad155cf4b9a940295ffbb17b6d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43488
I20260620 14:23:05.586396 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet c83a5ad155cf4b9a940295ffbb17b6d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59694
I20260620 14:23:05.586647 14800 tablet_service.cc:1514] Processing DeleteTablet for tablet c83a5ad155cf4b9a940295ffbb17b6d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37476
I20260620 14:23:05.586943 15172 tablet_replica.cc:330] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: stopping tablet replica
I20260620 14:23:05.587193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 14716
I20260620 14:23:05.587255 15172 raft_consensus.cc:2229] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:23:05.587755 15172 raft_consensus.cc:2258] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:05.588517 15171 tablet_replica.cc:330] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: stopping tablet replica
I20260620 14:23:05.588835 15171 raft_consensus.cc:2229] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:05.589140 15171 raft_consensus.cc:2258] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:05.589164 15172 ts_tablet_manager.cc:1905] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:05.590608 15171 ts_tablet_manager.cc:1905] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:05.592757 15172 ts_tablet_manager.cc:1918] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:05.592854 15172 log.cc:1196] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/c83a5ad155cf4b9a940295ffbb17b6d0
I20260620 14:23:05.593206 15172 ts_tablet_manager.cc:1939] T c83a5ad155cf4b9a940295ffbb17b6d0 P aa7ea15fb6f143d5889105b47f8ac0fe: Deleting consensus metadata
I20260620 14:23:05.593219 15171 ts_tablet_manager.cc:1918] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:05.593299 15171 log.cc:1196] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/c83a5ad155cf4b9a940295ffbb17b6d0
I20260620 14:23:05.593581 15171 ts_tablet_manager.cc:1939] T c83a5ad155cf4b9a940295ffbb17b6d0 P db35da665df64cf28fd01a2aadeb90e8: Deleting consensus metadata
I20260620 14:23:05.594372 14663 catalog_manager.cc:4841] TS aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087): tablet c83a5ad155cf4b9a940295ffbb17b6d0 (table test-table [id=9081cdd415d34a3da5cc38e66f263a65]) successfully deleted
I20260620 14:23:05.594614 14664 catalog_manager.cc:4841] TS db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097): tablet c83a5ad155cf4b9a940295ffbb17b6d0 (table test-table [id=9081cdd415d34a3da5cc38e66f263a65]) successfully deleted
W20260620 14:23:05.596007 14662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45805 (error 108)
I20260620 14:23:05.596999 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.596706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45805 (local address 127.0.0.1:37486)
0620 14:23:05.596828 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:05.596899 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:23:05.597118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 14870
W20260620 14:23:05.597226 14662 catalog_manager.cc:4584] TS 758afbb641df4c788857c4d8f345c6d4 (127.0.98.1:45805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c83a5ad155cf4b9a940295ffbb17b6d0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45805: connect: Connection refused (error 111)
I20260620 14:23:05.605038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 15007
I20260620 14:23:05.610283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 14645
I20260620 14:23:05.616842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40601
--webserver_interface=127.0.98.62
--webserver_port=33901
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:05.728868 15175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:05.729152 15175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:05.729195 15175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:05.733428 15175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:05.733486 15175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:05.733506 15175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:05.733523 15175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:05.733565 15175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:05.739030 15175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:05.740173 15175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:05.741245 15175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:05.750344 15183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:05.750344 15181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:05.750353 15180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:05.750571 15175 server_base.cc:1034] running on GCE node
I20260620 14:23:05.751179 15175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:05.752403 15175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:05.753593 15175 hybrid_clock.cc:648] HybridClock initialized: now 1781965385753575 us; error 28 us; skew 500 ppm
Jun 20 14:23:05 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:05.756706 15175 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:05.757607 15175 webserver.cc:460] Webserver started at http://127.0.98.62:33901/ using document root <none> and password file <none>
I20260620 14:23:05.758167 15175 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:05.758234 15175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:05.762105 15175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:23:05.764277 15190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:05.765564 15175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:23:05.765722 15175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "2b33de6e3cbe4bf99328ca8e61362ddf"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "16744b4359d17d030d67ba33e29aed53"
server_key_iv: "3f3c67fb204e7786dc762d4bf4165bfb"
server_key_version: "encryptionkey@0"
I20260620 14:23:05.766278 15175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:05.790685 15175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:05.793885 15175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:05.794106 15175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:05.801393 15175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40601
I20260620 14:23:05.801674 15242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40601 every 8 connection(s)
I20260620 14:23:05.802492 15175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:05.803123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 15175
I20260620 14:23:05.803570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45805
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40601
--webserver_port=41039
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:05.807395 15243 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:23:05.809108 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Bootstrap starting.
I20260620 14:23:05.826236 15243 log.cc:824] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:05.837540 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:23:05.837930 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf: Bootstrap complete.
I20260620 14:23:05.841940 15243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:05.842357 15243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b33de6e3cbe4bf99328ca8e61362ddf, State: Initialized, Role: FOLLOWER
I20260620 14:23:05.842873 15243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:05.843020 15243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:05.843106 15243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:05.843221 15243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:23:05.845115 15243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:05.845444 15243 leader_election.cc:302] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b33de6e3cbe4bf99328ca8e61362ddf; no voters:
I20260620 14:23:05.845783 15243 leader_election.cc:288] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:23:05.845865 15247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:23:05.846041 15247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [term 3 LEADER]: Becoming Leader. State: Replica: 2b33de6e3cbe4bf99328ca8e61362ddf, State: Running, Role: LEADER
I20260620 14:23:05.846335 15247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:23:05.846994 15243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:05.847584 15248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } } }
I20260620 14:23:05.847705 15248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:05.847957 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b33de6e3cbe4bf99328ca8e61362ddf. Latest consensus state: current_term: 3 leader_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33de6e3cbe4bf99328ca8e61362ddf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } } }
I20260620 14:23:05.848057 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33de6e3cbe4bf99328ca8e61362ddf [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:05.848901 15256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:05.853850 15256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:05.854538 15256 catalog_manager.cc:1259] Loaded cluster ID: 67b3a0ddab0d4b0faedf87f44026ea02
I20260620 14:23:05.854619 15256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:23:05.855492 15256 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f57912489ad google::LogMessage::Fail() at ??:0
@ 0x7f579124b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f57912484de google::LogMessage::Flush() at ??:0
@ 0x7f5791249a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f579bb87fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f579bb88d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f579bb85579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f579bbc9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f579349a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f579349ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f579349c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f579348b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5794acf6db start_thread at ??:0
@ 0x7f578fd4661f clone at ??:0
W20260620 14:23:05.945595 15245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:05.945855 15245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:05.945899 15245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:05.950059 15245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:05.950122 15245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:05.950237 15245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:23:05.955623 15245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41039
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:05.956754 15245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:05.957823 15245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:05.967357 15284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:05.967391 15285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:05.967576 15245 server_base.cc:1034] running on GCE node
W20260620 14:23:05.967809 15287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:05.968303 15245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:05.969120 15245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:05.970348 15245 hybrid_clock.cc:648] HybridClock initialized: now 1781965385970323 us; error 31 us; skew 500 ppm
Jun 20 14:23:05 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:05.973665 15245 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:23:05.974534 15245 webserver.cc:460] Webserver started at http://127.0.98.1:41039/ using document root <none> and password file <none>
I20260620 14:23:05.975108 15245 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:05.975178 15245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:05.979087 15245 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:23:05.981441 15294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:05.982817 15245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:23:05.982970 15245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "758afbb641df4c788857c4d8f345c6d4"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "5555730b768911cfb2202403b863bb8d"
server_key_iv: "9b74683a6817b6cedb346a2a8494675d"
server_key_version: "encryptionkey@0"
I20260620 14:23:05.983546 15245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:05.996409 15245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:05.999662 15245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:05.999897 15245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:06.000505 15245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:06.001585 15301 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:23:06.006938 15245 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:23:06.007030 15245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 14:23:06.007130 15245 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:23:06.008813 15245 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:23:06.008888 15245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 14:23:06.009205 15301 tablet_bootstrap.cc:492] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Bootstrap starting.
I20260620 14:23:06.019647 15245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45805
I20260620 14:23:06.019836 15408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45805 every 8 connection(s)
I20260620 14:23:06.020646 15245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:23:06.022199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 15245
I20260620 14:23:06.022676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35097
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40601
--webserver_port=34987
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:06.031810 15301 log.cc:824] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:06.040778 15301 tablet_bootstrap.cc:492] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:23:06.041179 15301 tablet_bootstrap.cc:492] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Bootstrap complete.
I20260620 14:23:06.042610 15301 ts_tablet_manager.cc:1397] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.004s
I20260620 14:23:06.045751 15301 raft_consensus.cc:348] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:06.046273 15301 raft_consensus.cc:729] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758afbb641df4c788857c4d8f345c6d4, State: Initialized, Role: FOLLOWER
I20260620 14:23:06.046772 15301 consensus_queue.cc:260] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:06.047806 15301 ts_tablet_manager.cc:1428] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260620 14:23:06.069670 15411 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:06.022735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:36947)
0620 14:23:06.024211 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:06.024276 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:06.025013 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:06.068850 (+ 43837us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1218,"thread_start_us":127,"threads_started":1}
W20260620 14:23:06.070168 15409 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40601 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40601: Client connection negotiation failed: client connection to 127.0.98.62:40601: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:23:06.070883 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.070641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:33205)
0620 14:23:06.070759 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:06.070833 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:23:06.071509 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.071295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:46649)
0620 14:23:06.071404 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:06.071448 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:23:06.071664 15409 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:23:06.144500 15412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:06.144815 15412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:06.144891 15412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:06.149066 15412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:06.149149 15412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:06.149247 15412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:23:06.154731 15412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34987
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:06.155890 15412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:06.156947 15412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:06.170812 15419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:06.171239 15420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:06.170812 15422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:06.171837 15412 server_base.cc:1034] running on GCE node
I20260620 14:23:06.172413 15412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:06.173549 15412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:06.174754 15412 hybrid_clock.cc:648] HybridClock initialized: now 1781965386174733 us; error 62 us; skew 500 ppm
Jun 20 14:23:06 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:06.178128 15412 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:23:06.179348 15412 webserver.cc:460] Webserver started at http://127.0.98.2:34987/ using document root <none> and password file <none>
I20260620 14:23:06.180107 15412 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:06.180197 15412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:06.185401 15412 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:23:06.188337 15429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:06.190125 15412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:23:06.190264 15412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "db35da665df64cf28fd01a2aadeb90e8"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "c64a992725791071fc7d1f86181c2d09"
server_key_iv: "a67b146bd78fd620b92c18d49124ea46"
server_key_version: "encryptionkey@0"
I20260620 14:23:06.190984 15412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:06.218662 15412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:06.221916 15412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:06.222132 15412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:06.222775 15412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:06.223624 15412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:06.223706 15412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:06.223826 15412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:06.223871 15412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:06.233481 15412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35097
I20260620 14:23:06.233632 15542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35097 every 8 connection(s)
I20260620 14:23:06.234550 15412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:23:06.238448 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.236798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:59587)
0620 14:23:06.237498 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:06.237612 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":114,"threads_started":1}
W20260620 14:23:06.238962 15543 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40601 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40601: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
I20260620 14:23:06.239598 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.239360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:57845)
0620 14:23:06.239467 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:06.239544 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:23:06.240229 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.240035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:50031)
0620 14:23:06.240137 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:06.240177 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:23:06.240415 15543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:23:06.240520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 15412
I20260620 14:23:06.240933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34087
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40601
--webserver_port=41453
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:06.351824 15546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:06.352067 15546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:06.352144 15546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:06.356140 15546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:06.356194 15546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:06.356251 15546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:23:06.361351 15546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41453
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:06.362362 15546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:06.363389 15546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:06.372963 15554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:06.373008 15551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:06.372982 15552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:06.373046 15546 server_base.cc:1034] running on GCE node
I20260620 14:23:06.373633 15546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:06.374258 15546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:06.375449 15546 hybrid_clock.cc:648] HybridClock initialized: now 1781965386375434 us; error 28 us; skew 500 ppm
Jun 20 14:23:06 dist-test-slave-87nc krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:06.378304 15546 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:23:06.379170 15546 webserver.cc:460] Webserver started at http://127.0.98.3:41453/ using document root <none> and password file <none>
I20260620 14:23:06.379745 15546 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:06.379894 15546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:06.383545 15546 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:23:06.386075 15561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:06.387441 15546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:23:06.387564 15546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "aa7ea15fb6f143d5889105b47f8ac0fe"
format_stamp: "Formatted at 2026-06-20 14:23:03 on dist-test-slave-87nc"
server_key: "49b9f84c94f3367f3bd30dc41183bc42"
server_key_iv: "ae8f4c0eeb24e14a60fe87a05b6363d1"
server_key_version: "encryptionkey@0"
I20260620 14:23:06.388192 15546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:06.408140 15546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:06.411329 15546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:06.411556 15546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:06.412256 15546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:06.413118 15546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:06.413192 15546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:06.413265 15546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:06.413311 15546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:06.422914 15546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34087
I20260620 14:23:06.422937 15674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34087 every 8 connection(s)
I20260620 14:23:06.424002 15546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:23:06.427515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 15546
I20260620 14:23:06.428058 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.426157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:42851)
0620 14:23:06.426612 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:06.427172 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
W20260620 14:23:06.428539 15675 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40601 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40601: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
I20260620 14:23:06.429116 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.428888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:37969)
0620 14:23:06.429012 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:06.429060 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:23:06.429757 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.429552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:41767)
0620 14:23:06.429656 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:06.429709 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:23:06.429924 15675 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:23:07.072844 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:07.072352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:49573)
0620 14:23:07.072646 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:07.072728 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
I20260620 14:23:07.241353 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:07.240954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:45853)
0620 14:23:07.241192 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:07.241247 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260620 14:23:07.430992 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:07.430561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:34435)
0620 14:23:07.430814 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:07.430873 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260620 14:23:07.730600 15682 raft_consensus.cc:482] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:07.730854 15682 raft_consensus.cc:504] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:07.732060 15682 leader_election.cc:288] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097), aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087)
Jun 20 14:23:07 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 14:23:07 dist-test-slave-87nc krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:23:07.746325 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:07.732439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50341 (local address 127.0.98.2:35097)
0620 14:23:07.732846 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:07.732876 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:07.732925 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:23:07.733716 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:07.733723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:07.734174 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:07.734469 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:07.735631 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:07.736695 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:07.737876 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:07.738322 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:07.738495 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:07.741569 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:07.741583 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:07.741595 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:07.741624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:07.743413 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:07.744081 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:07.744086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:07.744090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:07.744258 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:07.744660 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:07.744665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:07.744666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:07.745197 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:07.745364 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:07.745893 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 14:23:07.746146 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":59,"threads_started":1}
I20260620 14:23:07.746325 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:07.732282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35097 (local address 127.0.98.1:50341)
0620 14:23:07.732588 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:07.732604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:07.732721 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:07.734527 (+ 1806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:07.734533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:07.735030 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:07.735462 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:07.735478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:07.736843 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:07.736848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:07.737745 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:07.737753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:07.738446 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:07.738450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:07.738705 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:07.739353 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:07.739369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:07.741352 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:07.743541 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:07.743547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:07.743561 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:07.743967 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:07.744371 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:07.744374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:07.744376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:07.744555 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:07.745337 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:07.745344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:07.745593 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 14:23:07.745785 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 14:23:07.746123 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":213,"spinlock_wait_cycles":41472,"thread_start_us":70,"threads_started":1}
I20260620 14:23:07.746366 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:07.732349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34087 (local address 127.0.98.1:50749)
0620 14:23:07.732622 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:07.732638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:07.732753 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:07.734527 (+ 1774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:07.734533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:07.735058 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:07.735462 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:07.735478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:07.736852 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:07.736858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:07.737746 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:07.737754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:07.738442 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:07.738445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:07.738702 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:07.739404 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:07.739416 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:07.741027 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:07.743799 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:07.743802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:07.743804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:07.744055 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:07.744403 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:07.744406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:07.744408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:07.744568 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:07.745343 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:07.745346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:07.745593 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 14:23:07.745785 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 14:23:07.746207 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":206,"spinlock_wait_cycles":18816,"thread_start_us":57,"threads_started":1}
I20260620 14:23:07.747023 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:07.732629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50749 (local address 127.0.98.3:34087)
0620 14:23:07.732964 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:07.732989 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:07.733020 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:23:07.733744 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:07.733752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:07.734170 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:07.734469 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:07.735628 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:07.736733 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:07.737866 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:07.738320 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:07.738497 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:07.741729 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:07.741740 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:07.741764 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:07.741830 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:07.743659 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:07.744161 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:07.744168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:07.744173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:07.744292 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:07.744679 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:07.744682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:07.744684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:07.745188 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:07.745364 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:07.745891 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 14:23:07.746187 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":95,"threads_started":1}
W20260620 14:23:07.748932 15297 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097): Not found: Tablet not found: c83a5ad155cf4b9a940295ffbb17b6d0
W20260620 14:23:07.749092 15296 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087): Not found: Tablet not found: c83a5ad155cf4b9a940295ffbb17b6d0
I20260620 14:23:07.749191 15296 leader_election.cc:302] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 758afbb641df4c788857c4d8f345c6d4; no voters: aa7ea15fb6f143d5889105b47f8ac0fe, db35da665df64cf28fd01a2aadeb90e8
I20260620 14:23:07.749421 15682 raft_consensus.cc:2726] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:23:08.073887 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:08.073628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:42837)
0620 14:23:08.073753 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:08.073806 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:23:08.242393 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:08.242010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:42415)
0620 14:23:08.242253 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:08.242305 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260620 14:23:08.432500 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:08.431938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:42267)
0620 14:23:08.432277 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:08.432379 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260620 14:23:09.075109 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:09.074697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:58093)
0620 14:23:09.074955 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:09.075013 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260620 14:23:09.243713 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:09.243272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:53077)
0620 14:23:09.243550 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:09.243612 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260620 14:23:09.433727 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:09.433260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:43131)
0620 14:23:09.433541 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:09.433620 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260620 14:23:09.863539 15692 raft_consensus.cc:482] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:09.863695 15692 raft_consensus.cc:504] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:09.864183 15692 leader_election.cc:288] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097), aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087)
W20260620 14:23:09.865022 15297 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097): Not found: Tablet not found: c83a5ad155cf4b9a940295ffbb17b6d0
W20260620 14:23:09.865121 15296 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087): Not found: Tablet not found: c83a5ad155cf4b9a940295ffbb17b6d0
I20260620 14:23:09.865247 15296 leader_election.cc:302] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 758afbb641df4c788857c4d8f345c6d4; no voters: aa7ea15fb6f143d5889105b47f8ac0fe, db35da665df64cf28fd01a2aadeb90e8
I20260620 14:23:09.865428 15692 raft_consensus.cc:2726] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:23:10.076534 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:10.076072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:33487)
0620 14:23:10.076370 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:10.076433 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":76,"threads_started":1}
I20260620 14:23:10.244979 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:10.244549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:51707)
0620 14:23:10.244805 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:10.244872 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20260620 14:23:10.434949 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:10.434554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:49987)
0620 14:23:10.434797 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:10.434852 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260620 14:23:11.078013 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:11.077545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:56995)
0620 14:23:11.077856 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:11.077918 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":77,"threads_started":1}
I20260620 14:23:11.246290 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:11.245819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:54431)
0620 14:23:11.246107 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:11.246184 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":72,"threads_started":1}
I20260620 14:23:11.436106 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:11.435684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:57233)
0620 14:23:11.435960 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:11.436016 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":112,"threads_started":1}
I20260620 14:23:12.079360 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:12.078926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:40859)
0620 14:23:12.079199 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:12.079260 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260620 14:23:12.134711 15700 raft_consensus.cc:482] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:12.134899 15700 raft_consensus.cc:504] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:12.135354 15700 leader_election.cc:288] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097), aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087)
W20260620 14:23:12.136132 15297 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097): Not found: Tablet not found: c83a5ad155cf4b9a940295ffbb17b6d0
W20260620 14:23:12.136232 15296 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087): Not found: Tablet not found: c83a5ad155cf4b9a940295ffbb17b6d0
I20260620 14:23:12.136343 15296 leader_election.cc:302] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 758afbb641df4c788857c4d8f345c6d4; no voters: aa7ea15fb6f143d5889105b47f8ac0fe, db35da665df64cf28fd01a2aadeb90e8
I20260620 14:23:12.136500 15700 raft_consensus.cc:2726] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:23:12.247538 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:12.247123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:51813)
0620 14:23:12.247386 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:12.247443 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260620 14:23:12.437271 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:12.436862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:49941)
0620 14:23:12.437114 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:12.437175 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260620 14:23:13.080607 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:13.080199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:54039)
0620 14:23:13.080458 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:13.080512 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260620 14:23:13.248904 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:13.248406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:50971)
0620 14:23:13.248741 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:13.248807 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":95,"threads_started":1}
I20260620 14:23:13.438563 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:13.438079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:53651)
0620 14:23:13.438403 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:13.438468 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":73,"threads_started":1}
I20260620 14:23:14.081789 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:14.081394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:34503)
0620 14:23:14.081642 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:14.081694 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260620 14:23:14.250217 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:14.249753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:49463)
0620 14:23:14.250040 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:14.250104 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":123,"threads_started":1}
I20260620 14:23:14.439761 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:14.439360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:49231)
0620 14:23:14.439609 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:14.439669 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260620 14:23:15.083215 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:15.082750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:51909)
0620 14:23:15.083057 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:15.083119 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":87,"threads_started":1}
I20260620 14:23:15.251600 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:15.251139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:58173)
0620 14:23:15.251429 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:15.251504 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":113,"threads_started":1}
I20260620 14:23:15.440977 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:15.440562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:44587)
0620 14:23:15.440818 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:15.440871 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260620 14:23:16.084661 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:16.084218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:55701)
0620 14:23:16.084505 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:16.084564 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
I20260620 14:23:16.253149 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:16.252601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:57329)
0620 14:23:16.252903 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:16.252990 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260620 14:23:16.442157 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:16.441743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:33187)
0620 14:23:16.442006 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:16.442065 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260620 14:23:16.971694 15715 raft_consensus.cc:482] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:16.971891 15715 raft_consensus.cc:504] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db35da665df64cf28fd01a2aadeb90e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35097 } } peers { permanent_uuid: "aa7ea15fb6f143d5889105b47f8ac0fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34087 } } peers { permanent_uuid: "758afbb641df4c788857c4d8f345c6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45805 } }
I20260620 14:23:16.972384 15715 leader_election.cc:288] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097), aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087)
W20260620 14:23:16.973179 15297 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db35da665df64cf28fd01a2aadeb90e8 (127.0.98.2:35097): Not found: Tablet not found: c83a5ad155cf4b9a940295ffbb17b6d0
W20260620 14:23:16.973321 15296 leader_election.cc:341] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa7ea15fb6f143d5889105b47f8ac0fe (127.0.98.3:34087): Not found: Tablet not found: c83a5ad155cf4b9a940295ffbb17b6d0
I20260620 14:23:16.973412 15296 leader_election.cc:302] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 758afbb641df4c788857c4d8f345c6d4; no voters: aa7ea15fb6f143d5889105b47f8ac0fe, db35da665df64cf28fd01a2aadeb90e8
I20260620 14:23:16.973587 15715 raft_consensus.cc:2726] T c83a5ad155cf4b9a940295ffbb17b6d0 P 758afbb641df4c788857c4d8f345c6d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:23:17.086002 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:17.085508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:34851)
0620 14:23:17.085824 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:17.085889 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":107,"threads_started":1}
I20260620 14:23:17.254348 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:17.253951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:44063)
0620 14:23:17.254188 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:17.254254 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260620 14:23:17.443547 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:17.443079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:46161)
0620 14:23:17.443353 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:17.443453 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260620 14:23:18.087319 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:18.086838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:38643)
0620 14:23:18.087150 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:18.087215 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
I20260620 14:23:18.255465 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:18.255070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:35553)
0620 14:23:18.255316 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:18.255367 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260620 14:23:18.444849 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:18.444387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:49437)
0620 14:23:18.444684 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:18.444749 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
I20260620 14:23:19.088624 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:19.088156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:56331)
0620 14:23:19.088456 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:19.088524 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260620 14:23:19.256901 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:19.256449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:48801)
0620 14:23:19.256740 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:19.256803 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260620 14:23:19.446143 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:19.445715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:38295)
0620 14:23:19.445988 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:19.446049 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260620 14:23:20.089876 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:20.089432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:59205)
0620 14:23:20.089684 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:20.089758 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
I20260620 14:23:20.258215 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:20.257665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:35993)
0620 14:23:20.258032 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:20.258105 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":93,"threads_started":1}
I20260620 14:23:20.447371 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:20.446950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:60391)
0620 14:23:20.447219 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:20.447277 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260620 14:23:21.091420 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:21.090907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:55625)
0620 14:23:21.091242 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:21.091301 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":165,"threads_started":1}
I20260620 14:23:21.259624 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:21.259169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:60169)
0620 14:23:21.259471 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:21.259529 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":76,"threads_started":1}
I20260620 14:23:21.433804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 15245
I20260620 14:23:21.439850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 15412
I20260620 14:23:21.445226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 15546
2026-06-20T14:23:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19612 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:21 dist-test-slave-87nc krb5kdc[15732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:21 dist-test-slave-87nc krb5kdc[15732](info): set up 2 sockets
Jun 20 14:23:21 dist-test-slave-87nc krb5kdc[15732](info): commencing operation
krb5kdc: starting...
W20260620 14:23:22.492177 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:22 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:22.613950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:22.727236 15745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:22.727600 15745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:22.727691 15745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:22.731863 15745 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:23:22.731958 15745 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:23:22.731985 15745 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:23:22.732007 15745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:22.732049 15745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:22.732092 15745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:22.732128 15745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:22.732164 15745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:22.737600 15745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:22.738781 15745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:22.739936 15745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:22.748787 15750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:22.748781 15753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:22.748781 15751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:22.749068 15745 server_base.cc:1034] running on GCE node
I20260620 14:23:22.749933 15745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:22.751117 15745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:22.752296 15745 hybrid_clock.cc:648] HybridClock initialized: now 1781965402752261 us; error 47 us; skew 500 ppm
Jun 20 14:23:22 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:22.755216 15745 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:22.756302 15745 webserver.cc:460] Webserver started at http://127.0.98.62:33979/ using document root <none> and password file <none>
I20260620 14:23:22.756925 15745 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:22.757026 15745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:22.757277 15745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:22.759454 15745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "e19a35fd85d9462a98600ac1d08ec58a"
format_stamp: "Formatted at 2026-06-20 14:23:22 on dist-test-slave-87nc"
server_key: "f0491a91b2b588c43af14b47800b515d"
server_key_iv: "fad3901fbe4e5c58ba5fa88e8ee6c77d"
server_key_version: "encryptionkey@0"
I20260620 14:23:22.760094 15745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "e19a35fd85d9462a98600ac1d08ec58a"
format_stamp: "Formatted at 2026-06-20 14:23:22 on dist-test-slave-87nc"
server_key: "f0491a91b2b588c43af14b47800b515d"
server_key_iv: "fad3901fbe4e5c58ba5fa88e8ee6c77d"
server_key_version: "encryptionkey@0"
I20260620 14:23:22.764103 15745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:23:22.766641 15760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:22.767923 15745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:23:22.768067 15745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "e19a35fd85d9462a98600ac1d08ec58a"
format_stamp: "Formatted at 2026-06-20 14:23:22 on dist-test-slave-87nc"
server_key: "f0491a91b2b588c43af14b47800b515d"
server_key_iv: "fad3901fbe4e5c58ba5fa88e8ee6c77d"
server_key_version: "encryptionkey@0"
I20260620 14:23:22.768201 15745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:22.788717 15745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:22.791749 15745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:22.792037 15745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:22.799590 15745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39609
I20260620 14:23:22.799937 15812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39609 every 8 connection(s)
I20260620 14:23:22.800733 15745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:22.803534 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:22.809298 15813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Bootstrap starting.
I20260620 14:23:22.810460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 15745
I20260620 14:23:22.810565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:22.810889 392 external_mini_cluster.cc:1428] Setting key da6330bb989fa2ee10db616daa217b77
I20260620 14:23:22.812137 15813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:22.812894 15813 log.cc:824] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:22.815058 15813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: No bootstrap required, opened a new log
I20260620 14:23:22.819521 15813 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:23:22.819867 15813 raft_consensus.cc:374] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:22.819952 15813 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19a35fd85d9462a98600ac1d08ec58a, State: Initialized, Role: FOLLOWER
I20260620 14:23:22.820518 15813 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
Jun 20 14:23:22 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:22.820685 15813 raft_consensus.cc:388] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:22.820796 15813 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:22.820920 15813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:22.821877 15813 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:23:22.822238 15813 leader_election.cc:302] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e19a35fd85d9462a98600ac1d08ec58a; no voters:
I20260620 14:23:22.822567 15813 leader_election.cc:288] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:22.822731 15818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:22.822964 15818 raft_consensus.cc:686] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 1 LEADER]: Becoming Leader. State: Replica: e19a35fd85d9462a98600ac1d08ec58a, State: Running, Role: LEADER
I20260620 14:23:22.823338 15818 consensus_queue.cc:237] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:23:22.823923 15813 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:22.824688 15820 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e19a35fd85d9462a98600ac1d08ec58a. Latest consensus state: current_term: 1 leader_uuid: "e19a35fd85d9462a98600ac1d08ec58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } } }
I20260620 14:23:22.824822 15820 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:22.825543 15819 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e19a35fd85d9462a98600ac1d08ec58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } } }
I20260620 14:23:22.825706 15819 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:22.827487 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:22.812152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.0.98.62:39609)
0620 14:23:22.812644 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:22.812663 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:22.812700 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:23:22.813473 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:22.813492 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:22.813841 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:22.814139 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:22.815036 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:22.816292 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:22.817539 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:22.818041 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:22.818171 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:22.821356 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:22.821362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:22.821376 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:22.821407 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:22.823634 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:22.824185 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:22.824191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:22.824197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:22.824335 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:22.824672 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:22.824677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:22.824682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:22.825290 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:22.825492 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:22.826472 (+ 980us) server_negotiation.cc:299] Negotiation successful
0620 14:23:22.826758 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":176,"threads_started":1}
W20260620 14:23:22.829038 15833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:22.829113 15833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:22.829170 15826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:22.830026 15826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:22.835429 15826 catalog_manager.cc:1347] Generated new cluster ID: 032160f04c9a4b60a41e8dbc03faf539
I20260620 14:23:22.835511 15826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:22.862876 15826 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:22.863977 15826 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:22.868445 15826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Generated new TSK 0
I20260620 14:23:22.869273 15826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:23:22.904032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:23.020318 15839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:23.020566 15839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:23.020641 15839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:23.024741 15839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:23.024796 15839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:23.024912 15839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:23:23.030304 15839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:23.031368 15839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:23.032462 15839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:23.041880 15847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:23.041958 15844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:23.041880 15845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:23.042196 15839 server_base.cc:1034] running on GCE node
I20260620 14:23:23.042701 15839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:23.043358 15839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:23.044553 15839 hybrid_clock.cc:648] HybridClock initialized: now 1781965403044509 us; error 50 us; skew 500 ppm
Jun 20 14:23:23 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:23.047322 15839 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:23:23.048218 15839 webserver.cc:460] Webserver started at http://127.0.98.1:40571/ using document root <none> and password file <none>
I20260620 14:23:23.048746 15839 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:23.048806 15839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:23.048975 15839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:23.050968 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "39480b960640421c8c2f8e82f153e331"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "64639923befb55a276452717e3aa1d08"
server_key_iv: "7c164536f854dc192c8cb3996b46c70d"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.051466 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "39480b960640421c8c2f8e82f153e331"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "64639923befb55a276452717e3aa1d08"
server_key_iv: "7c164536f854dc192c8cb3996b46c70d"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.055135 15839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:23:23.057592 15854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.058804 15839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:23:23.058936 15839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "39480b960640421c8c2f8e82f153e331"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "64639923befb55a276452717e3aa1d08"
server_key_iv: "7c164536f854dc192c8cb3996b46c70d"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.059049 15839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:23.090458 15839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:23.093638 15839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:23.093798 15839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:23.094338 15839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:23.095140 15839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:23.095189 15839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.095228 15839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:23.095245 15839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.104528 15839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33041
I20260620 14:23:23.104548 15967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33041 every 8 connection(s)
I20260620 14:23:23.105451 15839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:23:23.111321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 15839
I20260620 14:23:23.111490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:23:23.111867 392 external_mini_cluster.cc:1428] Setting key 4e49b30994d17f885c6f0d3dc9803722
Jun 20 14:23:23 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:23.120874 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.107290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51263 (local address 127.0.98.62:39609)
0620 14:23:23.107455 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.107459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.108855 (+ 1396us) server_negotiation.cc:407] Connection header received
0620 14:23:23.109732 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.109734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.109781 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.109858 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:23.111082 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.111707 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.113259 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.113547 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.113633 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.116684 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:23.116688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:23.116691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:23.116716 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:23.118385 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:23.119094 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:23.119097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:23.119098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:23.119184 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:23.119617 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:23.119620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:23.119621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:23.119877 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:23.119958 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.120633 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.120745 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:23:23.122176 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.107676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:51263)
0620 14:23:23.108670 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:23.108727 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:23.109538 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:23.110033 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:23.110040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:23.110439 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:23.110942 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.110957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.111897 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.111902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.113135 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.113147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.113716 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.113723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.113957 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.114635 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:23.114665 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:23.116489 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:23.118518 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:23.118527 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:23.118548 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:23.118961 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:23.119284 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:23.119290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:23.119294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:23.119512 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:23.119986 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:23.119995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:23.120349 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 14:23:23.120601 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 14:23:23.120999 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":551,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:23:23.123557 15968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39609
I20260620 14:23:23.123912 15968 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:23.124588 15968 heartbeater.cc:511] Master 127.0.98.62:39609 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:23:23.126256 15777 ts_manager.cc:194] Registered new tserver with Master: 39480b960640421c8c2f8e82f153e331 (127.0.98.1:33041)
I20260620 14:23:23.127964 15777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:23:23.142401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:23.256556 15973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:23.256843 15973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:23.256888 15973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:23.260937 15973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:23.261008 15973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:23.261067 15973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:23:23.266602 15973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:23.267714 15973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:23.268793 15973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:23.278601 15981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:23.278622 15979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:23.278622 15978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:23.278672 15973 server_base.cc:1034] running on GCE node
I20260620 14:23:23.279233 15973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:23.279919 15973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:23.281109 15973 hybrid_clock.cc:648] HybridClock initialized: now 1781965403281088 us; error 33 us; skew 500 ppm
Jun 20 14:23:23 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:23.283874 15973 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:23:23.284731 15973 webserver.cc:460] Webserver started at http://127.0.98.2:36209/ using document root <none> and password file <none>
I20260620 14:23:23.285295 15973 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:23.285387 15973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:23.285616 15973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:23.287643 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "0514cf873bb34cb6906d0778c8481a10"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "8c83502b1d01aa04056ececbc28edae8"
server_key_iv: "0958f11d6919a559aad9f64c828ebe29"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.288305 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0514cf873bb34cb6906d0778c8481a10"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "8c83502b1d01aa04056ececbc28edae8"
server_key_iv: "0958f11d6919a559aad9f64c828ebe29"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.291975 15973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:23:23.294435 15988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.295699 15973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:23.295876 15973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "0514cf873bb34cb6906d0778c8481a10"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "8c83502b1d01aa04056ececbc28edae8"
server_key_iv: "0958f11d6919a559aad9f64c828ebe29"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.295997 15973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:23.317520 15973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:23.320567 15973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:23.320786 15973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:23.321405 15973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:23.322263 15973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:23.322337 15973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.322412 15973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:23.322455 15973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.331825 15973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35941
I20260620 14:23:23.331846 16101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35941 every 8 connection(s)
I20260620 14:23:23.332747 15973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:23:23.339184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 15973
I20260620 14:23:23.339310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:23:23.339614 392 external_mini_cluster.cc:1428] Setting key a6a97a01372b802e2f44e4e1e8a4f0c2
Jun 20 14:23:23 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:23.346899 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.334489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41975 (local address 127.0.98.62:39609)
0620 14:23:23.334658 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.334663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.335516 (+ 853us) server_negotiation.cc:407] Connection header received
0620 14:23:23.336801 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.336803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.336853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.336930 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:23.338098 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.338621 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.340205 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.340488 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.340643 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.343188 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:23.343191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:23.343194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:23.343231 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:23.344801 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:23.345350 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:23.345356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:23.345359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:23.345446 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:23.345790 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:23.345794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:23.345796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:23.345986 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:23.346096 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.346599 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.346728 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":70}
I20260620 14:23:23.347719 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.334877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:41975)
0620 14:23:23.335328 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:23.335370 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:23.336605 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:23.337092 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:23.337099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:23.337485 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:23.337972 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.337985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.338728 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.338731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.340087 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.340096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.340647 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.340650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.340784 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.341339 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:23.341355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:23.343004 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:23.344940 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:23.344946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:23.344962 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:23.345239 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:23.345540 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:23.345544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:23.345547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:23.345692 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:23.346097 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:23.346105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:23.346359 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 14:23:23.346550 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:23:23.346875 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:23:23.349170 16102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39609
I20260620 14:23:23.349436 16102 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:23.349967 16102 heartbeater.cc:511] Master 127.0.98.62:39609 requested a full tablet report, sending...
I20260620 14:23:23.351073 15777 ts_manager.cc:194] Registered new tserver with Master: 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:23:23.351934 15777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:23:23.366827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:23.477593 16107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:23.477855 16107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:23.477900 16107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:23.482091 16107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:23.482167 16107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:23.482260 16107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:23:23.488106 16107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:23.489214 16107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:23.490285 16107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:23.500159 16112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:23.500187 16113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:23.500167 16107 server_base.cc:1034] running on GCE node
W20260620 14:23:23.500219 16115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:23.500753 16107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:23.501435 16107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:23.502624 16107 hybrid_clock.cc:648] HybridClock initialized: now 1781965403502610 us; error 32 us; skew 500 ppm
Jun 20 14:23:23 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:23.505424 16107 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:23:23.506304 16107 webserver.cc:460] Webserver started at http://127.0.98.3:32887/ using document root <none> and password file <none>
I20260620 14:23:23.506923 16107 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:23.507016 16107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:23.507256 16107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:23.509438 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c9511ea9d3348a586e226af35e3ddd1"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "f7e7bffe570d66c518c8f3bcc9744184"
server_key_iv: "3683a82a1a0531cc51cfb60356c9c402"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.510033 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c9511ea9d3348a586e226af35e3ddd1"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "f7e7bffe570d66c518c8f3bcc9744184"
server_key_iv: "3683a82a1a0531cc51cfb60356c9c402"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.513999 16107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:23:23.516498 16122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.517702 16107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:23:23.517840 16107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "6c9511ea9d3348a586e226af35e3ddd1"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "f7e7bffe570d66c518c8f3bcc9744184"
server_key_iv: "3683a82a1a0531cc51cfb60356c9c402"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.517956 16107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:23.530730 16107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:23.534263 16107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:23.534482 16107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:23.535113 16107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:23.536034 16107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:23.536120 16107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.536194 16107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:23.536230 16107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.545506 16107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33545
I20260620 14:23:23.545527 16235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33545 every 8 connection(s)
I20260620 14:23:23.546424 16107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:23:23.553756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 16107
I20260620 14:23:23.553896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:23:23.554200 392 external_mini_cluster.cc:1428] Setting key ddcd95d47d274cef32e2d996e35e6bae
Jun 20 14:23:23 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:23.561559 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.548239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50933 (local address 127.0.98.62:39609)
0620 14:23:23.548374 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.548378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.549641 (+ 1263us) server_negotiation.cc:407] Connection header received
0620 14:23:23.550585 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.550588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.550644 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.550751 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:23.552008 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.552553 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.553734 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.554085 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.554203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.557146 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:23.557150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:23.557153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:23.557186 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:23.558722 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:23.559434 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:23.559437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:23.559439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:23.559562 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:23.560114 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:23.560118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:23.560123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:23.560387 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:23.560498 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.561257 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.561393 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:23:23.562597 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.548591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:50933)
0620 14:23:23.549470 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:23.549514 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:23.550360 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:23.550932 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:23.550940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:23.551304 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:23.551860 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.551877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.552685 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.552688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.553587 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.553597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.554245 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.554249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.554436 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.555140 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:23.555182 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:23.556951 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:23.558830 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:23.558835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:23.558850 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:23.559250 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:23.559711 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:23.559715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:23.559717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:23.559998 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:23.560526 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:23.560534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:23.560917 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 14:23:23.561202 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 14:23:23.561577 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":534,"threads_started":1}
I20260620 14:23:23.563733 16236 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39609
I20260620 14:23:23.564008 16236 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:23.564534 16236 heartbeater.cc:511] Master 127.0.98.62:39609 requested a full tablet report, sending...
I20260620 14:23:23.565527 15777 ts_manager.cc:194] Registered new tserver with Master: 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545)
I20260620 14:23:23.566102 15777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50933
I20260620 14:23:23.568344 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:23:23.578277 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.570264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55676 (local address 127.0.98.62:39609)
0620 14:23:23.570414 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.570424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.570522 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:23:23.570620 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.570622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.570659 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.570723 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:23.571355 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.572147 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.573301 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.573532 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.573638 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.575017 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:23.575021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:23.575023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:23.575049 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:23.576543 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:23.576982 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:23.576992 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:23.576994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:23.577066 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:23.577343 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:23.577348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:23.577349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:23.577573 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:23.577748 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.577996 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.578114 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:23:23.581207 15777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:23:23.584295 15777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:23:23.597930 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.592421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35941 (local address 127.0.0.1:54144)
0620 14:23:23.592806 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:23.592859 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:23.592996 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:23.593818 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:23.593825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:23.593850 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:23.594029 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.594038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.595395 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.595401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.596609 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.596621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.597381 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.597384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.597500 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.597536 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:23:23.597664 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:23:23.597787 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260620 14:23:23.597941 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.592575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.0.98.2:35941)
0620 14:23:23.593291 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.593305 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.593328 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:23:23.593409 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.593414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.593636 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.594186 (+ 550us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:23.594339 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.595271 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.596713 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.597258 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.597383 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.597532 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.597704 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.597795 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":136,"threads_started":1}
I20260620 14:23:23.598766 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.592620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33545 (local address 127.0.0.1:53598)
0620 14:23:23.593167 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:23.593182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:23.593268 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:23.594461 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:23.594464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:23.594481 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:23.594600 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.594608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.596132 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.596136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.597123 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.597132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.598406 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.598410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.598494 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.598517 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:23:23.598565 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:23:23.598634 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":35,"thread_start_us":91,"threads_started":1}
I20260620 14:23:23.599545 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.593758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.0.98.3:33545)
0620 14:23:23.594093 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.594102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.594120 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:23:23.594179 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.594183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.594325 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.594482 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:23.594726 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.595964 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.597259 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.598272 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.599093 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.599260 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.599312 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.599401 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":146,"threads_started":1}
I20260620 14:23:23.599974 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.593201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33041 (local address 127.0.0.1:36736)
0620 14:23:23.593899 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:23.593911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:23.593996 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:23.594674 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:23.594677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:23.594693 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:23.594771 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.594778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.596148 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.596151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.598260 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.598270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.599519 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.599523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.599653 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.599681 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:23:23.599749 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:23:23.599848 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260620 14:23:23.600436 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.593367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.0.98.1:33041)
0620 14:23:23.593767 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.593788 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.594091 (+ 303us) server_negotiation.cc:407] Connection header received
0620 14:23:23.594170 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.594175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.594365 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.594521 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:23.594902 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.596017 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.598402 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.599236 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.599339 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.599459 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.600208 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.600302 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":118,"threads_started":1}
I20260620 14:23:23.601357 16036 tablet_service.cc:1467] Processing CreateTablet for tablet 06369f8ef46145ffaa7990bbbf62a228 (DEFAULT_TABLE table=test-table [id=c73ce358665a42f386887c9d6972ef92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:23.602501 16036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06369f8ef46145ffaa7990bbbf62a228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:23.602468 16170 tablet_service.cc:1467] Processing CreateTablet for tablet 06369f8ef46145ffaa7990bbbf62a228 (DEFAULT_TABLE table=test-table [id=c73ce358665a42f386887c9d6972ef92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:23.603531 16170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06369f8ef46145ffaa7990bbbf62a228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:23.605533 15902 tablet_service.cc:1467] Processing CreateTablet for tablet 06369f8ef46145ffaa7990bbbf62a228 (DEFAULT_TABLE table=test-table [id=c73ce358665a42f386887c9d6972ef92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:23.606392 15902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06369f8ef46145ffaa7990bbbf62a228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:23.610108 16253 tablet_bootstrap.cc:492] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: Bootstrap starting.
I20260620 14:23:23.612080 16254 tablet_bootstrap.cc:492] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: Bootstrap starting.
I20260620 14:23:23.612663 16255 tablet_bootstrap.cc:492] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Bootstrap starting.
I20260620 14:23:23.613570 16253 tablet_bootstrap.cc:654] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:23.614244 16254 tablet_bootstrap.cc:654] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:23.614657 16253 log.cc:824] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:23.614820 16255 tablet_bootstrap.cc:654] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:23.615634 16254 log.cc:824] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:23.616113 16255 log.cc:824] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:23.616873 16253 tablet_bootstrap.cc:492] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: No bootstrap required, opened a new log
I20260620 14:23:23.617100 16253 ts_tablet_manager.cc:1397] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:23:23.617543 16254 tablet_bootstrap.cc:492] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: No bootstrap required, opened a new log
I20260620 14:23:23.617722 16254 ts_tablet_manager.cc:1397] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:23:23.618010 16255 tablet_bootstrap.cc:492] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: No bootstrap required, opened a new log
I20260620 14:23:23.618191 16255 ts_tablet_manager.cc:1397] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:23:23.620273 16253 raft_consensus.cc:348] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:23.620672 16253 raft_consensus.cc:374] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:23.620792 16253 raft_consensus.cc:729] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0514cf873bb34cb6906d0778c8481a10, State: Initialized, Role: FOLLOWER
I20260620 14:23:23.621416 16253 consensus_queue.cc:260] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:23.621620 16254 raft_consensus.cc:348] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:23.621945 16254 raft_consensus.cc:374] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:23.622005 16254 raft_consensus.cc:729] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9511ea9d3348a586e226af35e3ddd1, State: Initialized, Role: FOLLOWER
I20260620 14:23:23.622074 16255 raft_consensus.cc:348] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:23.622301 16255 raft_consensus.cc:374] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:23.622361 16255 raft_consensus.cc:729] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39480b960640421c8c2f8e82f153e331, State: Initialized, Role: FOLLOWER
I20260620 14:23:23.622421 16254 consensus_queue.cc:260] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:23.622663 16253 ts_tablet_manager.cc:1428] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:23:23.623070 16102 heartbeater.cc:503] Master 127.0.98.62:39609 was elected leader, sending a full tablet report...
I20260620 14:23:23.623219 16236 heartbeater.cc:503] Master 127.0.98.62:39609 was elected leader, sending a full tablet report...
I20260620 14:23:23.623502 16254 ts_tablet_manager.cc:1428] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:23:23.623571 16255 consensus_queue.cc:260] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:23.624285 15968 heartbeater.cc:503] Master 127.0.98.62:39609 was elected leader, sending a full tablet report...
I20260620 14:23:23.624682 16255 ts_tablet_manager.cc:1428] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:23:23.754604 16259 raft_consensus.cc:482] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:23.754812 16259 raft_consensus.cc:504] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:23.755903 16259 leader_election.cc:288] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545), 39480b960640421c8c2f8e82f153e331 (127.0.98.1:33041)
I20260620 14:23:23.761077 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.756658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52595 (local address 127.0.98.1:33041)
0620 14:23:23.756803 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.756810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.756960 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:23:23.757061 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.757064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.757107 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.757164 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:23.757421 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.758146 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.759766 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.760414 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.760526 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.760589 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.760886 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.760951 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:23:23.761077 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.756285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57481 (local address 127.0.98.3:33545)
0620 14:23:23.756436 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.756441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.756457 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:23:23.756672 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.756675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.756740 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.756813 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:23.757251 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.758060 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.759790 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.760414 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.760526 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.760588 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.760881 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.760951 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:23:23.761114 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.756204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33545 (local address 127.0.98.2:57481)
0620 14:23:23.756366 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:23.756383 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:23.756502 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:23.756938 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:23.756944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:23.756960 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:23.757104 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.757110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.758175 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.758178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.759616 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.759628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.760551 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.760554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.760734 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.760774 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:23:23.760847 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:23:23.760941 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":175}
I20260620 14:23:23.761121 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.756531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33041 (local address 127.0.98.2:52595)
0620 14:23:23.756874 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:23.756887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:23.756971 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:23.757199 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:23.757202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:23.757223 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:23.757319 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.757323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.758251 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.758254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.759617 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:23.759628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.760540 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.760544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:23.760701 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.760765 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:23:23.760847 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:23:23.760941 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":107,"thread_start_us":153,"threads_started":1}
I20260620 14:23:23.761819 15922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06369f8ef46145ffaa7990bbbf62a228" candidate_uuid: "0514cf873bb34cb6906d0778c8481a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39480b960640421c8c2f8e82f153e331" is_pre_election: true
I20260620 14:23:23.761819 16190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06369f8ef46145ffaa7990bbbf62a228" candidate_uuid: "0514cf873bb34cb6906d0778c8481a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9511ea9d3348a586e226af35e3ddd1" is_pre_election: true
I20260620 14:23:23.762130 15922 raft_consensus.cc:2454] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0514cf873bb34cb6906d0778c8481a10 in term 0.
I20260620 14:23:23.762171 16190 raft_consensus.cc:2454] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0514cf873bb34cb6906d0778c8481a10 in term 0.
I20260620 14:23:23.762635 15989 leader_election.cc:302] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0514cf873bb34cb6906d0778c8481a10, 39480b960640421c8c2f8e82f153e331; no voters:
I20260620 14:23:23.762930 16259 raft_consensus.cc:2781] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:23:23.763010 16259 raft_consensus.cc:482] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:23:23.763047 16259 raft_consensus.cc:3037] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:23.764178 16259 raft_consensus.cc:504] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:23.764554 16259 leader_election.cc:288] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [CANDIDATE]: Term 1 election: Requested vote from peers 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545), 39480b960640421c8c2f8e82f153e331 (127.0.98.1:33041)
I20260620 14:23:23.765025 15922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06369f8ef46145ffaa7990bbbf62a228" candidate_uuid: "0514cf873bb34cb6906d0778c8481a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39480b960640421c8c2f8e82f153e331"
I20260620 14:23:23.765025 16190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06369f8ef46145ffaa7990bbbf62a228" candidate_uuid: "0514cf873bb34cb6906d0778c8481a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9511ea9d3348a586e226af35e3ddd1"
I20260620 14:23:23.765158 16190 raft_consensus.cc:3037] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:23.765158 15922 raft_consensus.cc:3037] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:23.766436 15922 raft_consensus.cc:2454] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0514cf873bb34cb6906d0778c8481a10 in term 1.
I20260620 14:23:23.766436 16190 raft_consensus.cc:2454] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0514cf873bb34cb6906d0778c8481a10 in term 1.
I20260620 14:23:23.766748 15989 leader_election.cc:302] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0514cf873bb34cb6906d0778c8481a10, 39480b960640421c8c2f8e82f153e331; no voters:
I20260620 14:23:23.766904 16259 raft_consensus.cc:2781] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:23.767131 16259 raft_consensus.cc:686] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 1 LEADER]: Becoming Leader. State: Replica: 0514cf873bb34cb6906d0778c8481a10, State: Running, Role: LEADER
I20260620 14:23:23.767459 16259 consensus_queue.cc:237] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:23.770864 15777 catalog_manager.cc:5495] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0514cf873bb34cb6906d0778c8481a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } health_report { overall_health: UNKNOWN } } }
I20260620 14:23:23.786940 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.783634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.0.98.2:35941)
0620 14:23:23.783759 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.783810 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.783827 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:23:23.783951 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.783954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.784010 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.784107 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:23.784309 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.784793 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.785604 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.785807 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.785879 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.786121 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:23.786522 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:23.786614 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.786664 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.786711 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:23:23.792824 15922 raft_consensus.cc:1261] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Refusing update from remote peer 0514cf873bb34cb6906d0778c8481a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:23.792825 16190 raft_consensus.cc:1261] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [term 1 FOLLOWER]: Refusing update from remote peer 0514cf873bb34cb6906d0778c8481a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:23.793545 16259 consensus_queue.cc:1035] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:23.793764 16263 consensus_queue.cc:1035] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:23:23.797264 16237 tablet.cc:2369] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:23:23.802418 16268 mvcc.cc:205] Tried to move back new op lower bound from 7298930293928165376 to 7298930293834371072. Current Snapshot: MvccSnapshot[applied={T|T < 7298930293928165376}]
I20260620 14:23:23.810992 15775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:23.811151 15775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:23.813663 15775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Sending DeleteTablet for 3 replicas of tablet 06369f8ef46145ffaa7990bbbf62a228
I20260620 14:23:23.814278 16036 tablet_service.cc:1514] Processing DeleteTablet for tablet 06369f8ef46145ffaa7990bbbf62a228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54144
I20260620 14:23:23.814947 16276 tablet_replica.cc:330] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: stopping tablet replica
I20260620 14:23:23.815004 16170 tablet_service.cc:1514] Processing DeleteTablet for tablet 06369f8ef46145ffaa7990bbbf62a228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53598
I20260620 14:23:23.815320 16276 raft_consensus.cc:2229] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:23:23.815748 15902 tablet_service.cc:1514] Processing DeleteTablet for tablet 06369f8ef46145ffaa7990bbbf62a228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36736
I20260620 14:23:23.815819 16276 raft_consensus.cc:2258] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:23.816447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 15839
I20260620 14:23:23.817442 16276 ts_tablet_manager.cc:1905] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:23.819999 16277 tablet_replica.cc:330] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: stopping tablet replica
I20260620 14:23:23.820257 16276 ts_tablet_manager.cc:1918] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:23.820420 16276 log.cc:1196] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/06369f8ef46145ffaa7990bbbf62a228
I20260620 14:23:23.820398 16277 raft_consensus.cc:2229] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:23.820801 16277 raft_consensus.cc:2258] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:23.820822 16276 ts_tablet_manager.cc:1939] T 06369f8ef46145ffaa7990bbbf62a228 P 0514cf873bb34cb6906d0778c8481a10: Deleting consensus metadata
I20260620 14:23:23.822127 15764 catalog_manager.cc:4841] TS 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941): tablet 06369f8ef46145ffaa7990bbbf62a228 (table test-table [id=c73ce358665a42f386887c9d6972ef92]) successfully deleted
I20260620 14:23:23.822438 16277 ts_tablet_manager.cc:1905] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:23:23.823933 15761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33041 (error 108)
I20260620 14:23:23.824177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 15973
I20260620 14:23:23.825403 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.825076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33041 (local address 127.0.0.1:36742)
0620 14:23:23.825260 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:23.825334 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 14:23:23.825589 15761 catalog_manager.cc:4584] TS 39480b960640421c8c2f8e82f153e331 (127.0.98.1:33041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06369f8ef46145ffaa7990bbbf62a228: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33041: connect: Connection refused (error 111)
I20260620 14:23:23.829053 16277 ts_tablet_manager.cc:1918] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:23.829188 16277 log.cc:1196] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/06369f8ef46145ffaa7990bbbf62a228
I20260620 14:23:23.829563 16277 ts_tablet_manager.cc:1939] T 06369f8ef46145ffaa7990bbbf62a228 P 6c9511ea9d3348a586e226af35e3ddd1: Deleting consensus metadata
I20260620 14:23:23.830821 15761 catalog_manager.cc:4841] TS 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545): tablet 06369f8ef46145ffaa7990bbbf62a228 (table test-table [id=c73ce358665a42f386887c9d6972ef92]) successfully deleted
I20260620 14:23:23.831032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 16107
I20260620 14:23:23.836690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 15745
I20260620 14:23:43.843415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39609
--webserver_interface=127.0.98.62
--webserver_port=33979
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:43.954221 16280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:43.954495 16280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:43.954536 16280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:43.958491 16280 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:23:43.958549 16280 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:23:43.958566 16280 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:23:43.958606 16280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:43.958624 16280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:43.958642 16280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:43.958658 16280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:43.958673 16280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:43.963979 16280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:43.965090 16280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:43.966178 16280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:43.979023 16286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:43.979040 16287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:43.979053 16289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:43.979434 16280 server_base.cc:1034] running on GCE node
I20260620 14:23:43.980103 16280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:43.981240 16280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:43.982441 16280 hybrid_clock.cc:648] HybridClock initialized: now 1781965423982414 us; error 30 us; skew 500 ppm
Jun 20 14:23:43 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:43.990586 16280 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:43.991566 16280 webserver.cc:460] Webserver started at http://127.0.98.62:33979/ using document root <none> and password file <none>
I20260620 14:23:43.992254 16280 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:43.992374 16280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:43.996358 16280 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:23:43.998752 16296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:44.000216 16280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:23:44.000355 16280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "e19a35fd85d9462a98600ac1d08ec58a"
format_stamp: "Formatted at 2026-06-20 14:23:22 on dist-test-slave-87nc"
server_key: "f0491a91b2b588c43af14b47800b515d"
server_key_iv: "fad3901fbe4e5c58ba5fa88e8ee6c77d"
server_key_version: "encryptionkey@0"
I20260620 14:23:44.000977 16280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:44.016139 16280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:44.019054 16280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:44.019263 16280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:44.026257 16280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39609
I20260620 14:23:44.026547 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39609 every 8 connection(s)
I20260620 14:23:44.027330 16280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:44.029992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 16280
I20260620 14:23:44.030447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33041
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39609
--webserver_port=40571
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:44.031739 16349 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:23:44.034198 16349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Bootstrap starting.
I20260620 14:23:44.055576 16349 log.cc:824] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:44.064003 16349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:23:44.064393 16349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Bootstrap complete.
I20260620 14:23:44.068548 16349 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:23:44.068984 16349 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19a35fd85d9462a98600ac1d08ec58a, State: Initialized, Role: FOLLOWER
I20260620 14:23:44.069509 16349 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:23:44.069626 16349 raft_consensus.cc:388] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:44.069711 16349 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:44.069830 16349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:23:44.071702 16349 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:23:44.072059 16349 leader_election.cc:302] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e19a35fd85d9462a98600ac1d08ec58a; no voters:
I20260620 14:23:44.072402 16349 leader_election.cc:288] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:23:44.072535 16353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:23:44.072806 16353 raft_consensus.cc:686] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 2 LEADER]: Becoming Leader. State: Replica: e19a35fd85d9462a98600ac1d08ec58a, State: Running, Role: LEADER
I20260620 14:23:44.073164 16353 consensus_queue.cc:237] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:23:44.073589 16349 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:44.074676 16355 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e19a35fd85d9462a98600ac1d08ec58a. Latest consensus state: current_term: 2 leader_uuid: "e19a35fd85d9462a98600ac1d08ec58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } } }
I20260620 14:23:44.074707 16354 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e19a35fd85d9462a98600ac1d08ec58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } } }
I20260620 14:23:44.074813 16354 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:44.074811 16355 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:44.079001 16368 catalog_manager.cc:1259] Loaded cluster ID: 032160f04c9a4b60a41e8dbc03faf539
I20260620 14:23:44.079072 16368 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: loading cluster ID for follower catalog manager: success
I20260620 14:23:44.081593 16368 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: acquiring CA information for follower catalog manager: success
I20260620 14:23:44.083170 16368 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:23:44.083320 16362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:44.085819 16362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:44.086428 16362 catalog_manager.cc:1259] Loaded cluster ID: 032160f04c9a4b60a41e8dbc03faf539
I20260620 14:23:44.086489 16362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:44.087172 16362 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:44.087673 16362 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Loaded TSK: 0
I20260620 14:23:44.104318 16362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Generated new TSK 1
I20260620 14:23:44.104619 16362 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:23:44.151185 16351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:44.151499 16351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:44.151577 16351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:44.155653 16351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:44.155761 16351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:44.155884 16351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:23:44.161396 16351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40571
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:44.162519 16351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:44.163578 16351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:44.173542 16377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:44.173527 16376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:44.173736 16351 server_base.cc:1034] running on GCE node
W20260620 14:23:44.173861 16379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:44.174335 16351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:44.175160 16351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:44.176337 16351 hybrid_clock.cc:648] HybridClock initialized: now 1781965424176310 us; error 52 us; skew 500 ppm
Jun 20 14:23:44 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:44.179379 16351 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:23:44.180437 16351 webserver.cc:460] Webserver started at http://127.0.98.1:40571/ using document root <none> and password file <none>
I20260620 14:23:44.181012 16351 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:44.181128 16351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:44.185318 16351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:23:44.187721 16386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:44.189139 16351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:23:44.189247 16351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "39480b960640421c8c2f8e82f153e331"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "64639923befb55a276452717e3aa1d08"
server_key_iv: "7c164536f854dc192c8cb3996b46c70d"
server_key_version: "encryptionkey@0"
I20260620 14:23:44.189863 16351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:44.213982 16351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:44.217504 16351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:44.217707 16351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:44.218376 16351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:44.219519 16393 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:23:44.223618 16351 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:23:44.223683 16351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 14:23:44.223729 16351 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:23:44.225453 16351 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:23:44.225510 16351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:23:44.225754 16393 tablet_bootstrap.cc:492] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Bootstrap starting.
I20260620 14:23:44.236393 16351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33041
I20260620 14:23:44.236402 16500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33041 every 8 connection(s)
I20260620 14:23:44.237569 16351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:23:44.237897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 16351
I20260620 14:23:44.238382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35941
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39609
--webserver_port=36209
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:44.248934 16393 log.cc:824] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:23:44 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:44.256283 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.241261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50699 (local address 127.0.98.62:39609)
0620 14:23:44.241636 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.241655 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.241701 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:23:44.242136 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.242147 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.242447 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.242768 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:44.243954 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.245416 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.246597 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.247070 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.247257 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.249999 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:44.250007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:44.250029 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:44.250075 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:44.252577 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.253242 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.253250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.253256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.253440 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.253821 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.253824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.253826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.254296 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:44.254427 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.255083 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.255404 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
I20260620 14:23:44.256301 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.240074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:50699)
0620 14:23:44.240697 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:44.240757 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:44.241714 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:44.242824 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:44.242832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:44.243249 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:44.243703 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.243717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.245592 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.245595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.246462 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.246472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.247223 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.247227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.247435 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.248106 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:44.248123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:44.249745 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:44.252720 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:44.252726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:44.252740 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:44.253086 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:44.253548 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:44.253551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:44.253553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:44.253713 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:44.254421 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:44.254428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:44.254727 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 14:23:44.254928 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 14:23:44.255259 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":168,"threads_started":1}
I20260620 14:23:44.257325 16393 tablet_bootstrap.cc:492] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:23:44.257797 16393 tablet_bootstrap.cc:492] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Bootstrap complete.
I20260620 14:23:44.259018 16393 ts_tablet_manager.cc:1397] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.004s
I20260620 14:23:44.259675 16501 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39609
I20260620 14:23:44.259967 16501 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:44.260515 16501 heartbeater.cc:511] Master 127.0.98.62:39609 requested a full tablet report, sending...
I20260620 14:23:44.261554 16393 raft_consensus.cc:348] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.262073 16393 raft_consensus.cc:729] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39480b960640421c8c2f8e82f153e331, State: Initialized, Role: FOLLOWER
I20260620 14:23:44.262530 16393 consensus_queue.cc:260] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.263157 16313 ts_manager.cc:194] Registered new tserver with Master: 39480b960640421c8c2f8e82f153e331 (127.0.98.1:33041)
I20260620 14:23:44.263491 16393 ts_tablet_manager.cc:1428] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260620 14:23:44.267138 16313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50699
I20260620 14:23:44.269810 16501 heartbeater.cc:503] Master 127.0.98.62:39609 was elected leader, sending a full tablet report...
Jun 20 14:23:44 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:23:44.277551 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.266427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33041 (local address 127.0.0.1:60696)
0620 14:23:44.266796 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:44.266839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:44.266958 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:44.267602 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:44.267608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:44.268123 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:44.268268 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.268276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.269581 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.269584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.270606 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.270618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.271410 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.271425 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.271543 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.272166 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:44.272178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:44.273525 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:44.275880 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:44.275891 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:44.275901 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:44.276122 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:44.276461 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:44.276463 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:44.276465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:44.276540 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:44.277031 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:44.277036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:44.277142 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 14:23:44.277262 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:23:44.277420 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":111,"threads_started":1}
I20260620 14:23:44.277688 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.266470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.0.98.1:33041)
0620 14:23:44.267161 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.267170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.267186 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:23:44.267277 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.267290 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.267460 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.267600 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:44.268406 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.269459 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.270746 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.271225 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.271339 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.273719 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:44.273728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:44.273739 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:44.273764 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:44.275699 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.276260 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.276273 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.276275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.276366 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.276622 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.276625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.276627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.276906 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:44.277032 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.277365 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.277533 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":94,"threads_started":1}
I20260620 14:23:44.278630 16435 tablet_service.cc:1514] Processing DeleteTablet for tablet 06369f8ef46145ffaa7990bbbf62a228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60696
I20260620 14:23:44.279482 16510 tablet_replica.cc:330] stopping tablet replica
I20260620 14:23:44.279831 16510 raft_consensus.cc:2229] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:44.280090 16510 raft_consensus.cc:2258] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:44.281196 16510 ts_tablet_manager.cc:1905] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:44.285732 16510 ts_tablet_manager.cc:1918] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:44.285838 16510 log.cc:1196] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal/wals/06369f8ef46145ffaa7990bbbf62a228
I20260620 14:23:44.286103 16510 ts_tablet_manager.cc:1939] T 06369f8ef46145ffaa7990bbbf62a228 P 39480b960640421c8c2f8e82f153e331: Deleting consensus metadata
I20260620 14:23:44.288192 16297 catalog_manager.cc:4841] TS 39480b960640421c8c2f8e82f153e331 (127.0.98.1:33041): tablet 06369f8ef46145ffaa7990bbbf62a228 (table test-table [id=c73ce358665a42f386887c9d6972ef92]) successfully deleted
W20260620 14:23:44.364593 16503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:44.364962 16503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:44.365037 16503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:44.369401 16503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:44.369496 16503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:44.369589 16503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:23:44.375090 16503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36209
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:44.376255 16503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:44.377290 16503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:44.387068 16515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:44.387115 16516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:44.387080 16518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:44.387460 16503 server_base.cc:1034] running on GCE node
I20260620 14:23:44.387949 16503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:44.388588 16503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:44.389770 16503 hybrid_clock.cc:648] HybridClock initialized: now 1781965424389749 us; error 44 us; skew 500 ppm
Jun 20 14:23:44 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:44.392927 16503 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:23:44.393850 16503 webserver.cc:460] Webserver started at http://127.0.98.2:36209/ using document root <none> and password file <none>
I20260620 14:23:44.394424 16503 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:44.394519 16503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:44.398393 16503 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:44.400466 16525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:44.401655 16503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:44.401773 16503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "0514cf873bb34cb6906d0778c8481a10"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "8c83502b1d01aa04056ececbc28edae8"
server_key_iv: "0958f11d6919a559aad9f64c828ebe29"
server_key_version: "encryptionkey@0"
I20260620 14:23:44.402303 16503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:44.414566 16503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:44.417944 16503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:44.418154 16503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:44.418745 16503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:44.419584 16503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:44.419631 16503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:44.419693 16503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:44.419724 16503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:44.428263 16503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35941
I20260620 14:23:44.428362 16638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35941 every 8 connection(s)
I20260620 14:23:44.429301 16503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:23:44.438493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 16503
Jun 20 14:23:44 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:44.439033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33545
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39609
--webserver_port=32887
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:44.443719 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.431087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57775 (local address 127.0.98.62:39609)
0620 14:23:44.431222 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.431226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.432275 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 14:23:44.433180 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.433184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.433251 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.433362 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:44.434555 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.435217 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.436451 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.436690 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.436803 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.439539 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:44.439542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:44.439544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:44.439572 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:44.441277 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.441784 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.441789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.441794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.441884 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.442313 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.442319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.442324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.442581 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:44.442688 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.443312 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.443473 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:23:44.444481 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.431420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:57775)
0620 14:23:44.432121 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:44.432162 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:44.432963 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:44.433547 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:44.433554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:44.433935 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:44.434401 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.434415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.435357 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.435361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.436326 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.436335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.436850 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.436853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.437017 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.437587 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:44.437604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:44.439369 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:44.441409 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:44.441414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:44.441426 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:44.441676 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:44.442017 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:44.442020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:44.442022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:44.442197 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:44.442689 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:44.442693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:44.442962 (+ 269us) client_negotiation.cc:769] Sending connection context
0620 14:23:44.443219 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 14:23:44.443509 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":112,"threads_started":1}
I20260620 14:23:44.445744 16639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39609
I20260620 14:23:44.446007 16639 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:44.446491 16639 heartbeater.cc:511] Master 127.0.98.62:39609 requested a full tablet report, sending...
I20260620 14:23:44.447574 16313 ts_manager.cc:194] Registered new tserver with Master: 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941)
I20260620 14:23:44.448253 16313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57775
W20260620 14:23:44.550889 16642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:44.551210 16642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:44.551288 16642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:44.555312 16642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:44.555383 16642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:44.555496 16642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:23:44.560909 16642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32887
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:23:44.562034 16642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:44.563107 16642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:44.572548 16650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:44.572538 16647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:44.572626 16648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:44.572837 16642 server_base.cc:1034] running on GCE node
I20260620 14:23:44.573374 16642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:44.574141 16642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:44.575325 16642 hybrid_clock.cc:648] HybridClock initialized: now 1781965424575294 us; error 49 us; skew 500 ppm
Jun 20 14:23:44 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:44.578177 16642 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:23:44.579023 16642 webserver.cc:460] Webserver started at http://127.0.98.3:32887/ using document root <none> and password file <none>
I20260620 14:23:44.579558 16642 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:44.579629 16642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:44.583488 16642 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:23:44.585741 16657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:44.587066 16642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:23:44.587236 16642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "6c9511ea9d3348a586e226af35e3ddd1"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "f7e7bffe570d66c518c8f3bcc9744184"
server_key_iv: "3683a82a1a0531cc51cfb60356c9c402"
server_key_version: "encryptionkey@0"
I20260620 14:23:44.587890 16642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:44.615365 16642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:44.619144 16642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:44.619385 16642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:44.620122 16642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:44.621049 16642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:44.621120 16642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:44.621199 16642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:44.621241 16642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:44.633114 16642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33545
I20260620 14:23:44.633144 16770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33545 every 8 connection(s)
I20260620 14:23:44.634594 16642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:23:44.640081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 16642
Jun 20 14:23:44 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:44.650419 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.636657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54047 (local address 127.0.98.62:39609)
0620 14:23:44.636807 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.636817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.637752 (+ 935us) server_negotiation.cc:407] Connection header received
0620 14:23:44.638741 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.638746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.638808 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.638897 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:44.640206 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.640812 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.642584 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.642922 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.643022 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.645902 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:44.645905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:44.645907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:44.645928 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:44.647568 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.648194 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.648198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.648200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.648294 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.648743 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.648746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.648748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.649017 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:44.649105 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.649768 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.649906 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:23:44.650580 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.640737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.0.98.62:39609)
0620 14:23:44.640976 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.640979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.641133 (+ 154us) server_negotiation.cc:407] Connection header received
0620 14:23:44.641295 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.641297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.641333 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.641408 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:44.642369 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.643064 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.644244 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.644474 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.644551 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.645998 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:44.646002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:44.646005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:44.646052 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:44.648180 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.648719 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.648722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.648724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.648815 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.649122 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.649125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.649127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.649322 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:44.649520 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.650176 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.650298 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260620 14:23:44.651224 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.636953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:54047)
0620 14:23:44.637589 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:44.637634 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:44.638519 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:44.639079 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:44.639086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:44.639469 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:44.640032 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.640048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.640995 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.640999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.642471 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.642481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.643080 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.643084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.643313 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.643995 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:44.644015 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:44.645721 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:44.647692 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:44.647698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:44.647712 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:44.648072 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:44.648420 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:44.648423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:44.648425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:44.648622 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:44.649157 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:44.649166 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:44.649469 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 14:23:44.649671 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 14:23:44.650159 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":112,"threads_started":1}
I20260620 14:23:44.652403 16771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39609
I20260620 14:23:44.652679 16771 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:44.653194 16771 heartbeater.cc:511] Master 127.0.98.62:39609 requested a full tablet report, sending...
I20260620 14:23:44.654095 16312 ts_manager.cc:194] Registered new tserver with Master: 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545)
I20260620 14:23:44.654826 16312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54047
I20260620 14:23:44.664763 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:23:44.668625 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.665392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.0.98.62:39609)
0620 14:23:44.665524 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.665527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.665716 (+ 189us) server_negotiation.cc:407] Connection header received
0620 14:23:44.665865 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.665867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.665968 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.666054 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:44.666250 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.666744 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.667542 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.667735 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.667906 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.668175 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:44.668326 (+ 151us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:23:44.668524 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41046: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 14:23:44.668708 16505 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41046: token expired
W20260620 14:23:44.669030 16242 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39609)
I20260620 14:23:44.675951 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.669400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.0.98.62:39609)
0620 14:23:44.669519 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.669522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.669533 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:23:44.669594 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.669596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.669630 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.669692 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:44.670323 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.670773 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.671477 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.671667 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.671736 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.673046 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:44.673049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:44.673051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:44.673069 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:44.674461 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.674873 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.674876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.674877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.674945 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:44.675204 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:44.675206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:44.675207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:44.675369 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:44.675512 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.675679 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.675838 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:23:44.678144 16312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:23:44.681430 16312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:23:44.689543 16435 tablet_service.cc:1467] Processing CreateTablet for tablet bb2acee580aa450f98865358b7ff8b06 (DEFAULT_TABLE table=test-table [id=4dbbf33301264ea6826e0cc3b417748e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:44.690243 16435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2acee580aa450f98865358b7ff8b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:44.693244 16393 tablet_bootstrap.cc:492] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Bootstrap starting.
I20260620 14:23:44.694197 16393 tablet_bootstrap.cc:654] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:44.694617 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.688786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.0.98.3:33545)
0620 14:23:44.689265 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.689276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.689300 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:23:44.689374 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.689379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.689574 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.689909 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:44.690082 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.691245 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.693099 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.693959 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.694086 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.694235 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.694390 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.694482 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":169,"threads_started":1}
I20260620 14:23:44.694609 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.688566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33545 (local address 127.0.0.1:57800)
0620 14:23:44.689033 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:44.689046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:44.689117 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:44.689715 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:44.689718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:44.689736 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:44.689822 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.689828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.691357 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.691359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.692972 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.692983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.694088 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.694091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.694186 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.694231 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:23:44.694313 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:23:44.694399 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":121,"thread_start_us":99,"threads_started":1}
I20260620 14:23:44.695389 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.688323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35941 (local address 127.0.0.1:55070)
0620 14:23:44.688440 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:44.688457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:44.688538 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:44.689402 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:44.689405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:44.689421 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:44.689539 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.689546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.691332 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.691337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.692972 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.692983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.694962 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.694969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.695076 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.695099 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:23:44.695166 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:23:44.695254 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":188}
I20260620 14:23:44.695478 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.688497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.0.98.2:35941)
0620 14:23:44.688880 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.688897 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.688923 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:23:44.689018 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.689024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.689262 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.689429 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:44.689636 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.691178 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.693609 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.694431 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.694969 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.695130 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.695191 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.695293 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":135,"threads_started":1}
I20260620 14:23:44.695837 16393 tablet_bootstrap.cc:492] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: No bootstrap required, opened a new log
I20260620 14:23:44.695919 16393 ts_tablet_manager.cc:1397] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:23:44.696367 16393 raft_consensus.cc:348] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.696494 16393 raft_consensus.cc:374] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:44.696532 16393 raft_consensus.cc:729] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39480b960640421c8c2f8e82f153e331, State: Initialized, Role: FOLLOWER
I20260620 14:23:44.696656 16393 consensus_queue.cc:260] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.696902 16393 ts_tablet_manager.cc:1428] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:44.698462 16705 tablet_service.cc:1467] Processing CreateTablet for tablet bb2acee580aa450f98865358b7ff8b06 (DEFAULT_TABLE table=test-table [id=4dbbf33301264ea6826e0cc3b417748e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:44.699610 16705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2acee580aa450f98865358b7ff8b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:44.699661 16573 tablet_service.cc:1467] Processing CreateTablet for tablet bb2acee580aa450f98865358b7ff8b06 (DEFAULT_TABLE table=test-table [id=4dbbf33301264ea6826e0cc3b417748e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:44.700760 16573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2acee580aa450f98865358b7ff8b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:44.707965 16784 tablet_bootstrap.cc:492] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: Bootstrap starting.
I20260620 14:23:44.708953 16785 tablet_bootstrap.cc:492] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: Bootstrap starting.
I20260620 14:23:44.710362 16784 tablet_bootstrap.cc:654] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:44.711620 16785 tablet_bootstrap.cc:654] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:44.711706 16784 log.cc:824] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:44.712431 16785 log.cc:824] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:44.713801 16784 tablet_bootstrap.cc:492] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: No bootstrap required, opened a new log
I20260620 14:23:44.713995 16784 ts_tablet_manager.cc:1397] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:23:44.714334 16785 tablet_bootstrap.cc:492] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: No bootstrap required, opened a new log
I20260620 14:23:44.714506 16785 ts_tablet_manager.cc:1397] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:23:44.717962 16784 raft_consensus.cc:348] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.718274 16784 raft_consensus.cc:374] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:44.718331 16784 raft_consensus.cc:729] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9511ea9d3348a586e226af35e3ddd1, State: Initialized, Role: FOLLOWER
I20260620 14:23:44.718181 16785 raft_consensus.cc:348] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.718442 16785 raft_consensus.cc:374] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:44.718493 16785 raft_consensus.cc:729] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0514cf873bb34cb6906d0778c8481a10, State: Initialized, Role: FOLLOWER
I20260620 14:23:44.718851 16784 consensus_queue.cc:260] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.718851 16785 consensus_queue.cc:260] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.719507 16771 heartbeater.cc:503] Master 127.0.98.62:39609 was elected leader, sending a full tablet report...
I20260620 14:23:44.719748 16639 heartbeater.cc:503] Master 127.0.98.62:39609 was elected leader, sending a full tablet report...
I20260620 14:23:44.719863 16784 ts_tablet_manager.cc:1428] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:23:44.719863 16785 ts_tablet_manager.cc:1428] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260620 14:23:44.739071 16502 tablet.cc:2369] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:23:44.885838 16772 tablet.cc:2369] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:23:44.916654 16507 raft_consensus.cc:482] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:44.916923 16507 raft_consensus.cc:504] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.918112 16507 leader_election.cc:288] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941), 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545)
I20260620 14:23:44.922029 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.918449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32885 (local address 127.0.98.2:35941)
0620 14:23:44.918621 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.918626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.918746 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:23:44.918964 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.918969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.919023 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.919081 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:44.919377 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.919968 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.921071 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.921457 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.921577 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.921637 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.921828 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.921896 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:23:44.922043 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.918430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33545 (local address 127.0.98.1:46127)
0620 14:23:44.918759 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:44.918774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:44.918869 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:44.919162 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:44.919165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:44.919185 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:44.919297 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.919302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.920069 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.920073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.920905 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.920914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.921569 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.921572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.921668 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.921717 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 14:23:44.921808 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:23:44.921900 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260620 14:23:44.922034 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.918543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46127 (local address 127.0.98.3:33545)
0620 14:23:44.918720 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.918726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.918870 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:23:44.918997 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.918999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.919042 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.919133 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:44.919430 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.919966 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.921031 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.921446 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.921560 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.921622 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.921828 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.921905 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:23:44.922077 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.918312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35941 (local address 127.0.98.1:32885)
0620 14:23:44.918644 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:44.918660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:44.918799 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:44.919117 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:44.919123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:44.919141 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:44.919256 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.919278 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.920090 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.920094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.920933 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:44.920941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.921604 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.921607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:44.921702 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.921720 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:23:44.921812 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:23:44.921915 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":88,"thread_start_us":74,"threads_started":1}
I20260620 14:23:44.922845 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2acee580aa450f98865358b7ff8b06" candidate_uuid: "39480b960640421c8c2f8e82f153e331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9511ea9d3348a586e226af35e3ddd1" is_pre_election: true
I20260620 14:23:44.922884 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2acee580aa450f98865358b7ff8b06" candidate_uuid: "39480b960640421c8c2f8e82f153e331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0514cf873bb34cb6906d0778c8481a10" is_pre_election: true
I20260620 14:23:44.923167 16725 raft_consensus.cc:2454] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39480b960640421c8c2f8e82f153e331 in term 0.
I20260620 14:23:44.923183 16593 raft_consensus.cc:2454] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39480b960640421c8c2f8e82f153e331 in term 0.
I20260620 14:23:44.923661 16387 leader_election.cc:302] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0514cf873bb34cb6906d0778c8481a10, 39480b960640421c8c2f8e82f153e331, 6c9511ea9d3348a586e226af35e3ddd1; no voters:
I20260620 14:23:44.923971 16507 raft_consensus.cc:2781] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:23:44.924067 16507 raft_consensus.cc:482] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:23:44.924125 16507 raft_consensus.cc:3037] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:44.925202 16507 raft_consensus.cc:504] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:23:44.925607 16507 leader_election.cc:288] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 1 election: Requested vote from peers 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941), 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545)
I20260620 14:23:44.925946 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2acee580aa450f98865358b7ff8b06" candidate_uuid: "39480b960640421c8c2f8e82f153e331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0514cf873bb34cb6906d0778c8481a10"
I20260620 14:23:44.925992 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2acee580aa450f98865358b7ff8b06" candidate_uuid: "39480b960640421c8c2f8e82f153e331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9511ea9d3348a586e226af35e3ddd1"
I20260620 14:23:44.926095 16593 raft_consensus.cc:3037] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:44.926118 16725 raft_consensus.cc:3037] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:44.927069 16593 raft_consensus.cc:2454] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39480b960640421c8c2f8e82f153e331 in term 1.
I20260620 14:23:44.927093 16725 raft_consensus.cc:2454] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39480b960640421c8c2f8e82f153e331 in term 1.
I20260620 14:23:44.927377 16387 leader_election.cc:302] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39480b960640421c8c2f8e82f153e331, 6c9511ea9d3348a586e226af35e3ddd1; no voters:
I20260620 14:23:44.927569 16507 raft_consensus.cc:2781] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:44.927814 16507 raft_consensus.cc:686] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 LEADER]: Becoming Leader. State: Replica: 39480b960640421c8c2f8e82f153e331, State: Running, Role: LEADER
I20260620 14:23:44.928083 16507 consensus_queue.cc:237] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
W20260620 14:23:44.930404 16640 tablet.cc:2369] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:23:44.931216 16312 catalog_manager.cc:5495] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39480b960640421c8c2f8e82f153e331 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "39480b960640421c8c2f8e82f153e331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } health_report { overall_health: HEALTHY } } }
I20260620 14:23:44.990830 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:44.987322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.0.98.1:33041)
0620 14:23:44.987605 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:44.987612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:44.987625 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:23:44.987714 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:44.987717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:44.987834 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:44.987942 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:44.988159 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.988678 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.989441 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:44.989689 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:44.989766 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:44.990054 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:44.990498 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:44.990601 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:44.990659 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:23:44.990707 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":131,"threads_started":1}
I20260620 14:23:44.995041 16593 raft_consensus.cc:1261] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [term 1 FOLLOWER]: Refusing update from remote peer 39480b960640421c8c2f8e82f153e331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:44.995041 16725 raft_consensus.cc:1261] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [term 1 FOLLOWER]: Refusing update from remote peer 39480b960640421c8c2f8e82f153e331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:44.995755 16507 consensus_queue.cc:1035] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:44.996004 16792 consensus_queue.cc:1035] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:45.007761 16797 mvcc.cc:205] Tried to move back new op lower bound from 7298930380772909056 to 7298930380507832320. Current Snapshot: MvccSnapshot[applied={T|T < 7298930380772909056 or (T in {7298930380772909056})}]
I20260620 14:23:45.011337 16311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:45.011523 16311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:45.014006 16311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Sending DeleteTablet for 3 replicas of tablet bb2acee580aa450f98865358b7ff8b06
I20260620 14:23:45.014719 16573 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2acee580aa450f98865358b7ff8b06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55070
I20260620 14:23:45.014945 16435 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2acee580aa450f98865358b7ff8b06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60696
I20260620 14:23:45.014961 16705 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2acee580aa450f98865358b7ff8b06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57800
I20260620 14:23:45.015407 16808 tablet_replica.cc:330] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: stopping tablet replica
I20260620 14:23:45.015447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 16351
I20260620 14:23:45.016547 16808 raft_consensus.cc:2229] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:45.016896 16808 raft_consensus.cc:2258] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:45.017033 16806 tablet_replica.cc:330] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: stopping tablet replica
I20260620 14:23:45.017375 16806 raft_consensus.cc:2229] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:45.017693 16806 raft_consensus.cc:2258] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:45.018357 16808 ts_tablet_manager.cc:1905] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:45.019255 16806 ts_tablet_manager.cc:1905] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:45.020702 16808 ts_tablet_manager.cc:1918] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:45.020819 16808 log.cc:1196] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/bb2acee580aa450f98865358b7ff8b06
I20260620 14:23:45.021143 16808 ts_tablet_manager.cc:1939] T bb2acee580aa450f98865358b7ff8b06 P 6c9511ea9d3348a586e226af35e3ddd1: Deleting consensus metadata
I20260620 14:23:45.022326 16297 catalog_manager.cc:4841] TS 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545): tablet bb2acee580aa450f98865358b7ff8b06 (table test-table [id=4dbbf33301264ea6826e0cc3b417748e]) successfully deleted
I20260620 14:23:45.022629 16806 ts_tablet_manager.cc:1918] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:45.022758 16806 log.cc:1196] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/bb2acee580aa450f98865358b7ff8b06
I20260620 14:23:45.023097 16806 ts_tablet_manager.cc:1939] T bb2acee580aa450f98865358b7ff8b06 P 0514cf873bb34cb6906d0778c8481a10: Deleting consensus metadata
W20260620 14:23:45.023563 16297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33041 (error 108)
I20260620 14:23:45.024518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 16503
I20260620 14:23:45.024564 16300 catalog_manager.cc:4841] TS 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941): tablet bb2acee580aa450f98865358b7ff8b06 (table test-table [id=4dbbf33301264ea6826e0cc3b417748e]) successfully deleted
I20260620 14:23:45.025239 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:45.024314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33041 (local address 127.0.0.1:60726)
0620 14:23:45.024476 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:45.024552 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 14:23:45.025419 16297 catalog_manager.cc:4584] TS 39480b960640421c8c2f8e82f153e331 (127.0.98.1:33041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb2acee580aa450f98865358b7ff8b06: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33041: connect: Connection refused (error 111)
I20260620 14:23:45.030961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 16642
I20260620 14:23:45.039131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 16280
I20260620 14:24:05.046092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39609
--webserver_interface=127.0.98.62
--webserver_port=33979
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:05.158098 16810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:05.158380 16810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:05.158420 16810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:05.162582 16810 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:24:05.162650 16810 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:24:05.162667 16810 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:24:05.162681 16810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:05.162730 16810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:05.162770 16810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:05.162798 16810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:05.162829 16810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:05.168568 16810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:05.169879 16810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:05.170967 16810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:05.180240 16818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:05.180315 16816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:05.180240 16810 server_base.cc:1034] running on GCE node
W20260620 14:24:05.180294 16815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:05.181217 16810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:05.182437 16810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:05.183650 16810 hybrid_clock.cc:648] HybridClock initialized: now 1781965445183630 us; error 37 us; skew 500 ppm
Jun 20 14:24:05 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:05.189898 16810 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:05.190892 16810 webserver.cc:460] Webserver started at http://127.0.98.62:33979/ using document root <none> and password file <none>
I20260620 14:24:05.191529 16810 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:05.191637 16810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:05.195722 16810 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:05.197996 16825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:05.199261 16810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:05.199397 16810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "e19a35fd85d9462a98600ac1d08ec58a"
format_stamp: "Formatted at 2026-06-20 14:23:22 on dist-test-slave-87nc"
server_key: "f0491a91b2b588c43af14b47800b515d"
server_key_iv: "fad3901fbe4e5c58ba5fa88e8ee6c77d"
server_key_version: "encryptionkey@0"
I20260620 14:24:05.200040 16810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:05.213449 16810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:05.216624 16810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:05.216830 16810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:05.224231 16810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39609
I20260620 14:24:05.224548 16877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39609 every 8 connection(s)
I20260620 14:24:05.225383 16810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:05.229738 16878 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:24:05.231343 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Bootstrap starting.
I20260620 14:24:05.232995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 16810
I20260620 14:24:05.233495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33041
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39609
--webserver_port=40571
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:24:05.249383 16878 log.cc:824] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:05.260635 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:24:05.261029 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Bootstrap complete.
I20260620 14:24:05.265079 16878 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:24:05.265487 16878 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19a35fd85d9462a98600ac1d08ec58a, State: Initialized, Role: FOLLOWER
I20260620 14:24:05.265995 16878 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:24:05.266142 16878 raft_consensus.cc:388] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:05.266224 16878 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:05.266335 16878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:24:05.268352 16878 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:24:05.268707 16878 leader_election.cc:302] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e19a35fd85d9462a98600ac1d08ec58a; no voters:
I20260620 14:24:05.269026 16878 leader_election.cc:288] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:24:05.269163 16882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:24:05.269423 16882 raft_consensus.cc:686] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [term 3 LEADER]: Becoming Leader. State: Replica: e19a35fd85d9462a98600ac1d08ec58a, State: Running, Role: LEADER
I20260620 14:24:05.269760 16882 consensus_queue.cc:237] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } }
I20260620 14:24:05.270273 16878 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:05.270785 16882 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e19a35fd85d9462a98600ac1d08ec58a. Latest consensus state: current_term: 3 leader_uuid: "e19a35fd85d9462a98600ac1d08ec58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } } }
I20260620 14:24:05.270901 16882 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:05.271505 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e19a35fd85d9462a98600ac1d08ec58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19a35fd85d9462a98600ac1d08ec58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39609 } } }
I20260620 14:24:05.271606 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:05.272344 16887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:05.277884 16887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:05.278733 16887 catalog_manager.cc:1259] Loaded cluster ID: 032160f04c9a4b60a41e8dbc03faf539
I20260620 14:24:05.278807 16887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:05.281337 16887 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:05.282074 16887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Loaded TSK: 0
I20260620 14:24:05.282158 16887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e19a35fd85d9462a98600ac1d08ec58a: Loaded TSK: 1
F20260620 14:24:05.282867 16887 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f41a981b9ad google::LogMessage::Fail() at ??:0
@ 0x7f41a981e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f41a981b4de google::LogMessage::Flush() at ??:0
@ 0x7f41a981ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f41ac4df18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f41ac4d7f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f41b41807bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f41b415a99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f41b415b178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f41b419cdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f41b41bb82b std::function<>::operator()() at ??:0
@ 0x7f41b415ab4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f41b415bf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f41b4158579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f41b419c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f41aba6d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f41aba6dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f41aba6f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f41aba5e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f41ad0a26db start_thread at ??:0
@ 0x7f41a831961f clone at ??:0
W20260620 14:24:05.384298 16880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:05.384580 16880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:05.384625 16880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:05.388762 16880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:05.388828 16880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:05.388916 16880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:05.394503 16880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40571
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:05.395639 16880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:05.396710 16880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:05.407275 16926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:05.407501 16928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:05.407313 16925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:05.407853 16880 server_base.cc:1034] running on GCE node
I20260620 14:24:05.408252 16880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:05.408931 16880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:05.410106 16880 hybrid_clock.cc:648] HybridClock initialized: now 1781965445410094 us; error 32 us; skew 500 ppm
Jun 20 14:24:05 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:05.412899 16880 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:05.413903 16880 webserver.cc:460] Webserver started at http://127.0.98.1:40571/ using document root <none> and password file <none>
I20260620 14:24:05.414462 16880 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:05.414523 16880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:05.418545 16880 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:24:05.420806 16935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:05.422106 16880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:05.422227 16880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "39480b960640421c8c2f8e82f153e331"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "64639923befb55a276452717e3aa1d08"
server_key_iv: "7c164536f854dc192c8cb3996b46c70d"
server_key_version: "encryptionkey@0"
I20260620 14:24:05.422809 16880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:05.459180 16880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:05.462682 16880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:05.462888 16880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:05.463514 16880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:05.464715 16942 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:24:05.470530 16880 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:24:05.470721 16880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 14:24:05.470857 16880 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:24:05.472841 16880 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:24:05.472916 16880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260620 14:24:05.473229 16942 tablet_bootstrap.cc:492] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Bootstrap starting.
I20260620 14:24:05.483512 16880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33041
I20260620 14:24:05.483583 17049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33041 every 8 connection(s)
I20260620 14:24:05.484565 16880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:05.491164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 16880
I20260620 14:24:05.491644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35941
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39609
--webserver_port=36209
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:24:05.495692 16942 log.cc:824] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:05.505218 16942 tablet_bootstrap.cc:492] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:24:05.505668 16942 tablet_bootstrap.cc:492] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Bootstrap complete.
I20260620 14:24:05.507079 16942 ts_tablet_manager.cc:1397] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.004s
I20260620 14:24:05.510325 16942 raft_consensus.cc:348] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:24:05.510876 16942 raft_consensus.cc:729] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39480b960640421c8c2f8e82f153e331, State: Initialized, Role: FOLLOWER
I20260620 14:24:05.511404 16942 consensus_queue.cc:260] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:24:05.512528 16942 ts_tablet_manager.cc:1428] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260620 14:24:05.520236 17052 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:05.486550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:42171)
0620 14:24:05.487066 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:05.487125 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:05.488049 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:05.519261 (+ 31212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":156,"threads_started":1}
W20260620 14:24:05.520905 17050 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39609 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39609: Client connection negotiation failed: client connection to 127.0.98.62:39609: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:24:05.521639 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:05.521365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:44721)
0620 14:24:05.521499 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:05.521573 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:24:05.522291 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:05.522088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:33457)
0620 14:24:05.522190 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:05.522237 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:24:05.522534 17050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:24:05.608713 17053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:05.609004 17053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:05.609048 17053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:05.613271 17053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:05.613336 17053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:05.613425 17053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:05.618870 17053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36209
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:05.620128 17053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:05.621212 17053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:05.631039 17053 server_base.cc:1034] running on GCE node
W20260620 14:24:05.630986 17060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:05.631002 17063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:05.630988 17061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:05.631675 17053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:05.632344 17053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:05.633498 17053 hybrid_clock.cc:648] HybridClock initialized: now 1781965445633470 us; error 42 us; skew 500 ppm
Jun 20 14:24:05 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:05.636255 17053 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:05.637112 17053 webserver.cc:460] Webserver started at http://127.0.98.2:36209/ using document root <none> and password file <none>
I20260620 14:24:05.637642 17053 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:05.637706 17053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:05.641410 17053 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:05.643324 17070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:05.644550 17053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:24:05.644673 17053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "0514cf873bb34cb6906d0778c8481a10"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "8c83502b1d01aa04056ececbc28edae8"
server_key_iv: "0958f11d6919a559aad9f64c828ebe29"
server_key_version: "encryptionkey@0"
I20260620 14:24:05.645210 17053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:05.664969 17053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:05.668352 17053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:05.668560 17053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:05.669198 17053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:05.670089 17053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:05.670177 17053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:05.670248 17053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:05.670271 17053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:05.679394 17053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35941
I20260620 14:24:05.679414 17183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35941 every 8 connection(s)
I20260620 14:24:05.680477 17053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:05.684139 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:05.682606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:36529)
0620 14:24:05.683250 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:05.683353 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":101,"threads_started":1}
W20260620 14:24:05.684572 17184 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39609 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39609: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
I20260620 14:24:05.685120 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:05.684915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:35689)
0620 14:24:05.685009 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:05.685051 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:24:05.685635 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:05.685469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:35107)
0620 14:24:05.685555 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:05.685592 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:24:05.685796 17184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:24:05.690654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 17053
I20260620 14:24:05.691123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33545
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39609
--webserver_port=32887
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:05.802919 17187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:05.803191 17187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:05.803237 17187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:05.807197 17187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:05.807257 17187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:05.807317 17187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:05.812489 17187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32887
--tserver_master_addrs=127.0.98.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:05.813479 17187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:05.814515 17187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:05.824090 17193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:05.824108 17195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:05.824118 17192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:05.824453 17187 server_base.cc:1034] running on GCE node
I20260620 14:24:05.824937 17187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:05.825614 17187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:05.826805 17187 hybrid_clock.cc:648] HybridClock initialized: now 1781965445826769 us; error 48 us; skew 500 ppm
Jun 20 14:24:05 dist-test-slave-87nc krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:05.829552 17187 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:05.830436 17187 webserver.cc:460] Webserver started at http://127.0.98.3:32887/ using document root <none> and password file <none>
I20260620 14:24:05.830957 17187 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:05.831015 17187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:05.834865 17187 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:05.836905 17202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:05.838117 17187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:05.838209 17187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "6c9511ea9d3348a586e226af35e3ddd1"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-87nc"
server_key: "f7e7bffe570d66c518c8f3bcc9744184"
server_key_iv: "3683a82a1a0531cc51cfb60356c9c402"
server_key_version: "encryptionkey@0"
I20260620 14:24:05.838719 17187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:05.863031 17187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:05.865829 17187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:05.865991 17187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:05.866544 17187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:05.867376 17187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:05.867424 17187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:05.867492 17187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:05.867532 17187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:05.876959 17187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33545
I20260620 14:24:05.877048 17315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33545 every 8 connection(s)
I20260620 14:24:05.878026 17187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:05.881688 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:05.880190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:37621)
0620 14:24:05.880872 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:05.880977 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":114,"threads_started":1}
W20260620 14:24:05.882115 17316 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39609 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39609: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
I20260620 14:24:05.882722 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:05.882502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:48589)
0620 14:24:05.882619 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:05.882670 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:24:05.883301 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:05.883108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:42829)
0620 14:24:05.883202 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:05.883245 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:24:05.883459 17316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:24:05.887435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 17187
I20260620 14:24:06.523694 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:06.523216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:45121)
0620 14:24:06.523493 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:06.523559 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260620 14:24:06.686815 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:06.686331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:48469)
0620 14:24:06.686645 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:06.686706 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":68,"threads_started":1}
I20260620 14:24:06.884349 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:06.883981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:59999)
0620 14:24:06.884198 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:06.884248 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260620 14:24:07.107308 17323 raft_consensus.cc:482] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:07.107554 17323 raft_consensus.cc:504] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:24:07.108767 17323 leader_election.cc:288] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941), 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545)
Jun 20 14:24:07 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 14:24:07 dist-test-slave-87nc krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 14:24:07.123118 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:07.109031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33545 (local address 127.0.98.1:38817)
0620 14:24:07.109385 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:07.109399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:07.109493 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:07.111053 (+ 1560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:07.111058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:07.111616 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:07.112139 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:07.112154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:07.113611 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:07.113619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:07.114574 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:07.114584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:07.115286 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:07.115290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:07.115519 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:07.116257 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:07.116269 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:07.118041 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:07.120238 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:07.120241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:07.120243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:07.120550 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:07.120919 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:07.120922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:07.120924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:07.121157 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:07.121909 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:07.121916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:07.122300 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 14:24:07.122530 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 14:24:07.122948 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":352,"spinlock_wait_cycles":39296,"thread_start_us":78,"threads_started":1}
I20260620 14:24:07.123143 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:07.109191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43899 (local address 127.0.98.2:35941)
0620 14:24:07.109631 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:07.109655 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:07.109701 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:24:07.110365 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:07.110370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:07.110703 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:07.111005 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:07.112300 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:07.113484 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:07.114691 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:07.115145 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:07.115327 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:07.117890 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:07.117896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:07.117907 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:07.117934 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:07.119975 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:07.120653 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:07.120658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:07.120664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:07.120823 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:07.121255 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:07.121258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:07.121260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:07.121788 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:07.121927 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:07.122673 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 14:24:07.122959 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":81,"threads_started":1}
I20260620 14:24:07.123167 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:07.109226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38817 (local address 127.0.98.3:33545)
0620 14:24:07.109642 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:07.109665 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:07.109705 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:24:07.110364 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:07.110369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:07.110703 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:07.111005 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:07.112305 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:07.113471 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:07.114700 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:07.115144 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:07.115331 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:07.118240 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:07.118250 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:07.118266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:07.118299 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:07.120148 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:07.120653 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:07.120658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:07.120664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:07.120823 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:07.121255 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:07.121258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:07.121260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:07.121788 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:07.121927 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:07.122673 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 14:24:07.122959 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":104,"threads_started":1}
I20260620 14:24:07.123118 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:07.109010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35941 (local address 127.0.98.1:43899)
0620 14:24:07.109327 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:07.109342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:07.109457 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:07.111062 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:07.111066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:07.111606 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:07.112139 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:07.112153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:07.113613 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:07.113621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:07.114571 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:07.114580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:07.115261 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:07.115281 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:07.115518 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:07.116198 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:07.116214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:07.117670 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:07.120091 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:07.120096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:07.120108 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:07.120538 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:07.120919 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:07.120922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:07.120924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:07.121167 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:07.121909 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:07.121916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:07.122300 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 14:24:07.122531 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 14:24:07.122896 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":213,"spinlock_wait_cycles":15232,"thread_start_us":84,"threads_started":1}
W20260620 14:24:07.126341 16936 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
W20260620 14:24:07.126725 16939 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
I20260620 14:24:07.126833 16939 leader_election.cc:302] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39480b960640421c8c2f8e82f153e331; no voters: 0514cf873bb34cb6906d0778c8481a10, 6c9511ea9d3348a586e226af35e3ddd1
I20260620 14:24:07.127054 17323 raft_consensus.cc:2726] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:24:07.525002 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:07.524700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:54713)
0620 14:24:07.524845 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:07.524909 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:24:07.688158 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:07.687591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:51201)
0620 14:24:07.687978 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:07.688056 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":87,"threads_started":1}
I20260620 14:24:07.885406 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:07.885039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:57533)
0620 14:24:07.885271 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:07.885325 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260620 14:24:08.526194 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:08.525773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:34923)
0620 14:24:08.526021 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:08.526099 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260620 14:24:08.689275 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:08.688878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:34143)
0620 14:24:08.689111 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:08.689170 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260620 14:24:08.886564 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:08.886117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:50011)
0620 14:24:08.886361 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:08.886415 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260620 14:24:09.116611 17333 raft_consensus.cc:482] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:09.116756 17333 raft_consensus.cc:504] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:24:09.117182 17333 leader_election.cc:288] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941), 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545)
W20260620 14:24:09.118047 16939 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
W20260620 14:24:09.118152 16936 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
I20260620 14:24:09.118212 16936 leader_election.cc:302] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39480b960640421c8c2f8e82f153e331; no voters: 0514cf873bb34cb6906d0778c8481a10, 6c9511ea9d3348a586e226af35e3ddd1
I20260620 14:24:09.118350 17333 raft_consensus.cc:2726] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:24:09.527307 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:09.526891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:40333)
0620 14:24:09.527152 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:09.527212 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260620 14:24:09.690536 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:09.690093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:52039)
0620 14:24:09.690359 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:09.690440 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260620 14:24:09.887610 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:09.887244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:48621)
0620 14:24:09.887465 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:09.887517 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260620 14:24:10.528666 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:10.528214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:41683)
0620 14:24:10.528500 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:10.528571 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":70,"threads_started":1}
I20260620 14:24:10.692186 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:10.691648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:37325)
0620 14:24:10.691974 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:10.692046 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":113,"threads_started":1}
I20260620 14:24:10.888688 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:10.888326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:43141)
0620 14:24:10.888543 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:10.888595 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260620 14:24:10.928825 17340 raft_consensus.cc:482] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:10.928954 17340 raft_consensus.cc:504] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:24:10.929356 17340 leader_election.cc:288] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941), 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545)
W20260620 14:24:10.930128 16936 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
W20260620 14:24:10.930220 16939 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
I20260620 14:24:10.930331 16939 leader_election.cc:302] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39480b960640421c8c2f8e82f153e331; no voters: 0514cf873bb34cb6906d0778c8481a10, 6c9511ea9d3348a586e226af35e3ddd1
I20260620 14:24:10.930481 17340 raft_consensus.cc:2726] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:24:11.529870 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:11.529423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:35611)
0620 14:24:11.529693 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:11.529752 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260620 14:24:11.693315 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:11.692889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:45575)
0620 14:24:11.693171 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:11.693224 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":101,"threads_started":1}
I20260620 14:24:11.889816 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:11.889361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:49673)
0620 14:24:11.889657 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:11.889720 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
I20260620 14:24:12.530973 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:12.530583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:53727)
0620 14:24:12.530814 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:12.530868 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260620 14:24:12.694423 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:12.693958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:33481)
0620 14:24:12.694273 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:12.694330 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":85,"threads_started":1}
I20260620 14:24:12.891041 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:12.890583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:39773)
0620 14:24:12.890890 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:12.890948 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":80,"threads_started":1}
I20260620 14:24:13.532297 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:13.531840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:49339)
0620 14:24:13.532120 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:13.532194 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260620 14:24:13.695637 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:13.695187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:48689)
0620 14:24:13.695481 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:13.695542 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260620 14:24:13.892210 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:13.891809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:33021)
0620 14:24:13.892055 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:13.892114 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
I20260620 14:24:13.915987 17350 raft_consensus.cc:482] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:13.916117 17350 raft_consensus.cc:504] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:24:13.916596 17350 leader_election.cc:288] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941), 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545)
W20260620 14:24:13.917474 16939 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
W20260620 14:24:13.917613 16936 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
I20260620 14:24:13.917716 16936 leader_election.cc:302] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39480b960640421c8c2f8e82f153e331; no voters: 0514cf873bb34cb6906d0778c8481a10, 6c9511ea9d3348a586e226af35e3ddd1
I20260620 14:24:13.917891 17350 raft_consensus.cc:2726] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:24:14.533525 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:14.533144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:58077)
0620 14:24:14.533381 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:14.533434 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260620 14:24:14.696727 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:14.696342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:37553)
0620 14:24:14.696586 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:14.696638 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
I20260620 14:24:14.893350 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:14.892950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:58059)
0620 14:24:14.893201 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:14.893258 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260620 14:24:15.534863 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.534439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:52305)
0620 14:24:15.534712 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:15.534767 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
I20260620 14:24:15.697901 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.697512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:60093)
0620 14:24:15.697756 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:15.697808 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260620 14:24:15.894603 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.894192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:59015)
0620 14:24:15.894435 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:15.894488 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260620 14:24:16.536248 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:16.535669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:60331)
0620 14:24:16.536017 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:16.536110 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
I20260620 14:24:16.699237 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:16.698825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:44297)
0620 14:24:16.699080 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:16.699137 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260620 14:24:16.896006 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:16.895554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:39083)
0620 14:24:16.895859 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:16.895915 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260620 14:24:17.537403 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.537017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:58837)
0620 14:24:17.537253 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.537311 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260620 14:24:17.700516 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.700105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:44351)
0620 14:24:17.700358 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.700414 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260620 14:24:17.897188 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.896774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:33541)
0620 14:24:17.897029 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.897094 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260620 14:24:18.538751 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:18.538264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:56683)
0620 14:24:18.538584 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:18.538651 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
I20260620 14:24:18.701854 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:18.701394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:35603)
0620 14:24:18.701697 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:18.701760 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20260620 14:24:18.718149 17365 raft_consensus.cc:482] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:18.718298 17365 raft_consensus.cc:504] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0514cf873bb34cb6906d0778c8481a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35941 } } peers { permanent_uuid: "6c9511ea9d3348a586e226af35e3ddd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "39480b960640421c8c2f8e82f153e331" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33041 } }
I20260620 14:24:18.718845 17365 leader_election.cc:288] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941), 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545)
W20260620 14:24:18.719692 16939 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0514cf873bb34cb6906d0778c8481a10 (127.0.98.2:35941): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
W20260620 14:24:18.719826 16936 leader_election.cc:341] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c9511ea9d3348a586e226af35e3ddd1 (127.0.98.3:33545): Not found: Tablet not found: bb2acee580aa450f98865358b7ff8b06
I20260620 14:24:18.719897 16936 leader_election.cc:302] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39480b960640421c8c2f8e82f153e331; no voters: 0514cf873bb34cb6906d0778c8481a10, 6c9511ea9d3348a586e226af35e3ddd1
I20260620 14:24:18.720068 17365 raft_consensus.cc:2726] T bb2acee580aa450f98865358b7ff8b06 P 39480b960640421c8c2f8e82f153e331 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:24:18.898444 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:18.898001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:35249)
0620 14:24:18.898273 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:18.898334 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":69,"threads_started":1}
I20260620 14:24:19.540035 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.539553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:34193)
0620 14:24:19.539876 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.539935 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260620 14:24:19.703194 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.702704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:47013)
0620 14:24:19.703015 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.703080 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":80,"threads_started":1}
I20260620 14:24:19.899870 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.899289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:33797)
0620 14:24:19.899637 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.899705 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20260620 14:24:20.541484 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.541006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.1:37453)
0620 14:24:20.541314 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:20.541375 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":71,"threads_started":1}
I20260620 14:24:20.704537 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.704147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.2:41517)
0620 14:24:20.704390 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:20.704445 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260620 14:24:20.895887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 16880
I20260620 14:24:20.901436 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.900912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39609 (local address 127.0.98.3:58453)
0620 14:24:20.901208 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:20.901293 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":69,"threads_started":1}
I20260620 14:24:20.902156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 17053
I20260620 14:24:20.907660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 17187
2026-06-20T14:24:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59462 ms)
[----------] 30 tests from SecurityITest (246018 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:20 dist-test-slave-87nc krb5kdc[17375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:20 dist-test-slave-87nc krb5kdc[17375](info): set up 2 sockets
Jun 20 14:24:20 dist-test-slave-87nc krb5kdc[17375](info): commencing operation
krb5kdc: starting...
W20260620 14:24:21.939620 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:22 dist-test-slave-87nc krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:22.058233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46841
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:22.167367 17388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:22.167681 17388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:22.167754 17388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:22.171826 17388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:22.171928 17388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:22.171967 17388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:22.172022 17388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:22.172060 17388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:22.177604 17388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:22.178682 17388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:22.179716 17388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:22.188045 17394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:22.188216 17388 server_base.cc:1034] running on GCE node
W20260620 14:24:22.188074 17396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:22.188055 17393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:22.188877 17388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:22.189966 17388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:22.191192 17388 hybrid_clock.cc:648] HybridClock initialized: now 1781965462191161 us; error 41 us; skew 500 ppm
Jun 20 14:24:22 dist-test-slave-87nc krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:22.198760 17388 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:22.199744 17388 webserver.cc:460] Webserver started at http://127.0.98.62:42879/ using document root <none> and password file <none>
I20260620 14:24:22.200379 17388 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:22.200469 17388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:22.200697 17388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:22.202755 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "ca707caad7604acc96ad693fe9afdb3b"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "cfe39a44b884d761016ba2d64a5efe1d"
server_key_iv: "112bc0d19485082a0412f1c59723dd80"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.203351 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca707caad7604acc96ad693fe9afdb3b"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "cfe39a44b884d761016ba2d64a5efe1d"
server_key_iv: "112bc0d19485082a0412f1c59723dd80"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.207114 17388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260620 14:24:22.209604 17403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.210844 17388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:24:22.211002 17388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "ca707caad7604acc96ad693fe9afdb3b"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "cfe39a44b884d761016ba2d64a5efe1d"
server_key_iv: "112bc0d19485082a0412f1c59723dd80"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.211117 17388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:22.227577 17388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:22.231190 17388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:22.231428 17388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:22.238529 17388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46841
I20260620 14:24:22.238554 17455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46841 every 8 connection(s)
I20260620 14:24:22.239495 17388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:22.242197 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.244552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 17388
I20260620 14:24:22.244688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:22.244994 392 external_mini_cluster.cc:1428] Setting key e5c9b06e92aefd4b2b4188fc6074d437
I20260620 14:24:22.248536 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b: Bootstrap starting.
I20260620 14:24:22.250728 17456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.251464 17456 log.cc:824] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:22.253732 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b: No bootstrap required, opened a new log
Jun 20 14:24:22 dist-test-slave-87nc krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:22.257779 17456 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca707caad7604acc96ad693fe9afdb3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } }
I20260620 14:24:22.258111 17456 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.258205 17456 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca707caad7604acc96ad693fe9afdb3b, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.258718 17456 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca707caad7604acc96ad693fe9afdb3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } }
I20260620 14:24:22.258846 17456 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:22.258953 17456 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:22.259083 17456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.259553 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.246450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.0.98.62:46841)
0620 14:24:22.246877 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.246892 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.246935 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:24:22.247641 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.247658 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.248040 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.248383 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:22.249199 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.250205 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.251114 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.251617 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.251812 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.254638 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:22.254643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:22.254655 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:22.254682 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:22.256464 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.256965 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.256969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.256974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.257092 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.257378 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.257380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.257381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.257755 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:22.257901 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.258284 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.258608 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":129,"threads_started":1}
I20260620 14:24:22.260130 17456 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca707caad7604acc96ad693fe9afdb3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } }
I20260620 14:24:22.260479 17456 leader_election.cc:302] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca707caad7604acc96ad693fe9afdb3b; no voters:
I20260620 14:24:22.260850 17456 leader_election.cc:288] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:22.260931 17461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:22.261127 17461 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [term 1 LEADER]: Becoming Leader. State: Replica: ca707caad7604acc96ad693fe9afdb3b, State: Running, Role: LEADER
I20260620 14:24:22.261430 17461 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca707caad7604acc96ad693fe9afdb3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } }
I20260620 14:24:22.262145 17456 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:22.263338 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca707caad7604acc96ad693fe9afdb3b. Latest consensus state: current_term: 1 leader_uuid: "ca707caad7604acc96ad693fe9afdb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca707caad7604acc96ad693fe9afdb3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } }
I20260620 14:24:22.263471 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:22.263432 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca707caad7604acc96ad693fe9afdb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca707caad7604acc96ad693fe9afdb3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } }
I20260620 14:24:22.263541 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:22.263885 17470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:22.266961 17470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:22.272339 17470 catalog_manager.cc:1347] Generated new cluster ID: a21c704525ec44b2900e32a7a4fd75cb
I20260620 14:24:22.272423 17470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:22.296751 17470 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:22.297683 17470 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:22.303081 17470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b: Generated new TSK 0
I20260620 14:24:22.303730 17470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:22.339306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46841
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:22.449079 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:22.449337 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:22.449379 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:22.453369 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:22.453444 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:22.453502 17482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:22.458895 17482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:22.460042 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:22.461110 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:22.470997 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:22.471055 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:22.471167 17482 server_base.cc:1034] running on GCE node
W20260620 14:24:22.470997 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:22.471621 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:22.472327 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:22.473536 17482 hybrid_clock.cc:648] HybridClock initialized: now 1781965462473502 us; error 48 us; skew 500 ppm
Jun 20 14:24:22 dist-test-slave-87nc krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:22.476315 17482 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:22.477244 17482 webserver.cc:460] Webserver started at http://127.0.98.1:45417/ using document root <none> and password file <none>
I20260620 14:24:22.477818 17482 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:22.477910 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:22.478135 17482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:22.480247 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "d08638bffaa743e9a924a7d491c15480"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "221004d44d8286cd7989b93a08b65b56"
server_key_iv: "2a07b4bcf35ea8d5cdc90fa748d12bea"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.480816 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d08638bffaa743e9a924a7d491c15480"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "221004d44d8286cd7989b93a08b65b56"
server_key_iv: "2a07b4bcf35ea8d5cdc90fa748d12bea"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.484503 17482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:24:22.486857 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.488267 17482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:22.488430 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "d08638bffaa743e9a924a7d491c15480"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "221004d44d8286cd7989b93a08b65b56"
server_key_iv: "2a07b4bcf35ea8d5cdc90fa748d12bea"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.488554 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:22.509961 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:22.513599 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:22.513826 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:22.514521 17482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:22.515470 17482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:22.515538 17482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.515609 17482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:22.515655 17482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.524479 17482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46639
I20260620 14:24:22.524501 17610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46639 every 8 connection(s)
I20260620 14:24:22.525460 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:24:22 dist-test-slave-87nc krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:22.535705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 17482
I20260620 14:24:22.535868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:22.536163 392 external_mini_cluster.cc:1428] Setting key 083a2efe67a8ace753a39310229c717c
I20260620 14:24:22.538908 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.527274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41551 (local address 127.0.98.62:46841)
0620 14:24:22.527417 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.527421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.528293 (+ 872us) server_negotiation.cc:407] Connection header received
0620 14:24:22.529600 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.529603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.529661 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.529766 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:22.530893 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.531475 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.532696 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.532923 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.533008 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.535230 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:22.535233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:22.535235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:22.535257 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:22.536899 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.537395 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.537399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.537402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.537489 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.537866 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.537868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.537869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.538029 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:22.538100 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.538688 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.538779 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:24:22.539928 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.527682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46841 (local address 127.0.98.1:41551)
0620 14:24:22.528150 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.528193 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.529370 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.529936 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.529943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.530286 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:22.530749 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.530761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.531652 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.531655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.532586 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.532596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.533056 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.533059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.533196 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.533689 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:22.533704 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:22.535086 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:22.537016 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.537022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.537032 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.537296 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.537587 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.537590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.537592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.537774 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.538133 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:22.538141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:22.538456 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.538656 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.539006 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":113,"threads_started":1}
I20260620 14:24:22.541067 17611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46841
I20260620 14:24:22.541296 17611 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:22.541832 17611 heartbeater.cc:511] Master 127.0.98.62:46841 requested a full tablet report, sending...
I20260620 14:24:22.543521 17420 ts_manager.cc:194] Registered new tserver with Master: d08638bffaa743e9a924a7d491c15480 (127.0.98.1:46639)
I20260620 14:24:22.545120 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41551
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:22.562906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46841
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:22.676040 17616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:22.676282 17616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:22.676355 17616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:22.680353 17616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:22.680409 17616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:22.680514 17616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:22.685660 17616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:22.686717 17616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:22.687784 17616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:22.697624 17616 server_base.cc:1034] running on GCE node
W20260620 14:24:22.697585 17621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:22.697582 17622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:22.697585 17624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:22.698386 17616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:22.699056 17616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:22.700256 17616 hybrid_clock.cc:648] HybridClock initialized: now 1781965462700225 us; error 51 us; skew 500 ppm
Jun 20 14:24:22 dist-test-slave-87nc krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:22.702872 17616 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:22.703709 17616 webserver.cc:460] Webserver started at http://127.0.98.2:35111/ using document root <none> and password file <none>
I20260620 14:24:22.704310 17616 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:22.704401 17616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:22.704627 17616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:22.706619 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "d34f23077c2440028ca754af5d5400ff"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "83dbfc7fb8e867e780fefc00975f3f00"
server_key_iv: "b0decded98de687a973dbfa5c0bc1b7b"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.707193 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d34f23077c2440028ca754af5d5400ff"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "83dbfc7fb8e867e780fefc00975f3f00"
server_key_iv: "b0decded98de687a973dbfa5c0bc1b7b"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.710937 17616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:24:22.713325 17631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.714506 17616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:22.714653 17616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "d34f23077c2440028ca754af5d5400ff"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "83dbfc7fb8e867e780fefc00975f3f00"
server_key_iv: "b0decded98de687a973dbfa5c0bc1b7b"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.714793 17616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:22.735426 17616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:22.738392 17616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:22.738551 17616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:22.739180 17616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:22.740031 17616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:22.740079 17616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.740119 17616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:22.740141 17616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.748687 17616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38265
I20260620 14:24:22.748708 17744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38265 every 8 connection(s)
I20260620 14:24:22.749716 17616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:24:22 dist-test-slave-87nc krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:22.759383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 17616
I20260620 14:24:22.759524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:22.759893 392 external_mini_cluster.cc:1428] Setting key a9f1d65592c24dcdaad4d62abd75152a
I20260620 14:24:22.763991 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.751560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35115 (local address 127.0.98.62:46841)
0620 14:24:22.751734 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.751739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.752541 (+ 802us) server_negotiation.cc:407] Connection header received
0620 14:24:22.753855 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.753858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.753906 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.753991 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:22.755197 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.755707 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.756906 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.757130 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.757214 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.759808 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:22.759813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:22.759816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:22.759849 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:22.761498 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.762120 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.762123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.762124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.762196 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.762647 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.762649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.762650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.762815 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:22.762942 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.763624 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.763786 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:24:22.764833 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.751892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46841 (local address 127.0.98.2:35115)
0620 14:24:22.752384 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.752433 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.753649 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.754165 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.754172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.754532 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:22.755060 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.755075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.755879 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.755883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.756801 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.756810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.757261 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.757263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.757404 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.757964 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:22.757980 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:22.759593 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:22.761606 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.761613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.761631 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.762015 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.762350 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.762355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.762360 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.762554 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.762955 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:22.762964 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:22.763370 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.763556 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.763927 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260620 14:24:22.766084 17745 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46841
I20260620 14:24:22.766320 17745 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:22.766816 17745 heartbeater.cc:511] Master 127.0.98.62:46841 requested a full tablet report, sending...
I20260620 14:24:22.767874 17420 ts_manager.cc:194] Registered new tserver with Master: d34f23077c2440028ca754af5d5400ff (127.0.98.2:38265)
I20260620 14:24:22.768465 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35115
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:22.786474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46841
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:22.898352 17750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:22.898631 17750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:22.898676 17750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:22.902787 17750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:22.902853 17750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:22.902957 17750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:22.908411 17750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:22.909520 17750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:22.910585 17750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:22.920528 17758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:22.920641 17750 server_base.cc:1034] running on GCE node
W20260620 14:24:22.920531 17756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:22.920574 17755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:22.921217 17750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:22.921860 17750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:22.923053 17750 hybrid_clock.cc:648] HybridClock initialized: now 1781965462923015 us; error 48 us; skew 500 ppm
Jun 20 14:24:22 dist-test-slave-87nc krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:22.925796 17750 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:22.926671 17750 webserver.cc:460] Webserver started at http://127.0.98.3:40047/ using document root <none> and password file <none>
I20260620 14:24:22.927232 17750 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:22.927321 17750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:22.927549 17750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:22.929728 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "c0e65c3b45a04613bfc1f623ea97eec4"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "5f4e06a14978fa6f49459af3976492c0"
server_key_iv: "b828362f3fbd58a05d78ce12acacf660"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.930351 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c0e65c3b45a04613bfc1f623ea97eec4"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "5f4e06a14978fa6f49459af3976492c0"
server_key_iv: "b828362f3fbd58a05d78ce12acacf660"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.934185 17750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 14:24:22.936554 17765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.937680 17750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:22.937810 17750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "c0e65c3b45a04613bfc1f623ea97eec4"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-87nc"
server_key: "5f4e06a14978fa6f49459af3976492c0"
server_key_iv: "b828362f3fbd58a05d78ce12acacf660"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.937945 17750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:22.950560 17750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:22.953608 17750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:22.953830 17750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:22.954447 17750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:22.955294 17750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:22.955366 17750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.955432 17750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:22.955472 17750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.964421 17750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39179
I20260620 14:24:22.964491 17878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39179 every 8 connection(s)
I20260620 14:24:22.965337 17750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:22.973380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 17750
I20260620 14:24:22.973551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:22.973861 392 external_mini_cluster.cc:1428] Setting key 75642c8b6352d045636fb0d9bd4eb8ea
Jun 20 14:24:22 dist-test-slave-87nc krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:22.978480 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.967133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47363 (local address 127.0.98.62:46841)
0620 14:24:22.967274 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.967284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.968102 (+ 818us) server_negotiation.cc:407] Connection header received
0620 14:24:22.969351 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.969355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.969413 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.969526 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:22.970665 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.971199 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.972355 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.972582 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.972708 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.975436 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:22.975438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:22.975441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:22.975462 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:22.976885 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.977292 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.977294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.977296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.977362 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.977637 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.977639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.977640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.977791 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:22.977861 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.978287 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.978372 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:24:22.979278 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.967486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46841 (local address 127.0.98.3:47363)
0620 14:24:22.967956 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.968002 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.969147 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.969703 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.969710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.970046 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:22.970517 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.970531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.971323 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.971327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.972222 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.972231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.972745 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.972749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.972910 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.973531 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:22.973554 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:22.975297 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:22.976968 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.976972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.976983 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.977211 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.977435 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.977437 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.977438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.977563 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.977869 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:22.977874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:22.978101 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.978243 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.978464 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260620 14:24:22.980357 17879 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46841
I20260620 14:24:22.980579 17879 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:22.980969 17879 heartbeater.cc:511] Master 127.0.98.62:46841 requested a full tablet report, sending...
I20260620 14:24:22.981845 17420 ts_manager.cc:194] Registered new tserver with Master: c0e65c3b45a04613bfc1f623ea97eec4 (127.0.98.3:39179)
I20260620 14:24:22.982362 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47363
I20260620 14:24:22.987443 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:22.996156 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.989319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.0.98.62:46841)
0620 14:24:22.989468 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.989473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.989581 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:24:22.989673 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.989675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.989711 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.989773 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:22.990395 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.990875 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.991652 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.991898 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.991986 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.993285 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:22.993293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:22.993297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:22.993325 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:22.994728 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.995089 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.995092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.995093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.995161 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.995386 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.995388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.995389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.995570 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:22.995663 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.995882 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.995971 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:24:22.998899 17420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:23.001959 17420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:23.016032 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.011129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.0.98.2:38265)
0620 14:24:23.011390 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:23.011401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:23.011426 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:24:23.011521 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:23.011527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:23.011724 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:23.012116 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:23.012395 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.013137 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.014706 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.015340 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.015471 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.015621 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:23.015767 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:24:23.015874 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260620 14:24:23.016028 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.009995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38265 (local address 127.0.0.1:48206)
0620 14:24:23.010855 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:23.010868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:23.010958 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:23.011907 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:23.011910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:23.011941 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:23.012281 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.012287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.013274 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.013278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.014576 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.014587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.015467 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.015471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.015577 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.015612 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:24:23.015718 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:24:23.015874 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":116,"thread_start_us":87,"threads_started":1}
I20260620 14:24:23.016587 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.010621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39179 (local address 127.0.0.1:33366)
0620 14:24:23.011041 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:23.011054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:23.011136 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:23.011890 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:23.011896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:23.011919 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:23.012170 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.012180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.013731 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.013734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.014802 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.014811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.016011 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.016015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.016292 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.016318 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:24:23.016362 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:24:23.016439 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":57,"thread_start_us":118,"threads_started":1}
I20260620 14:24:23.017349 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.010819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.0.98.3:39179)
0620 14:24:23.011382 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:23.011394 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:23.011419 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:24:23.011514 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:23.011522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:23.011704 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:23.011893 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:23.012312 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.013534 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.014918 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.015885 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.016763 (+ 878us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.017057 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:23.017107 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:24:23.017195 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":125,"threads_started":1}
I20260620 14:24:23.017657 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.009925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46639 (local address 127.0.0.1:42804)
0620 14:24:23.010248 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:23.010294 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:23.010421 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:23.012077 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:23.012080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:23.012094 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:23.012296 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.012305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.013731 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.013734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.016031 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.016038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.017248 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.017251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.017353 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.017382 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:24:23.017460 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:24:23.017527 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":68,"thread_start_us":104,"threads_started":1}
I20260620 14:24:23.018119 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.010633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42804 (local address 127.0.98.1:46639)
0620 14:24:23.011528 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:23.011545 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:23.011564 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:24:23.011626 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:23.011630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:23.011767 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:23.011924 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:23.012435 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.013560 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.016153 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.016967 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.017068 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.017188 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:23.017859 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 14:24:23.017974 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":104,"threads_started":1}
I20260620 14:24:23.019490 17679 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd337546987437885f5287ba5543424 (DEFAULT_TABLE table=test-table [id=27cad5cbc399469897c9f00e781ffb19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:23.020478 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd337546987437885f5287ba5543424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:23.020210 17813 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd337546987437885f5287ba5543424 (DEFAULT_TABLE table=test-table [id=27cad5cbc399469897c9f00e781ffb19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:23.020736 17545 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd337546987437885f5287ba5543424 (DEFAULT_TABLE table=test-table [id=27cad5cbc399469897c9f00e781ffb19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:23.021215 17813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd337546987437885f5287ba5543424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:23.021631 17545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd337546987437885f5287ba5543424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:23.026638 17896 tablet_bootstrap.cc:492] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: Bootstrap starting.
I20260620 14:24:23.027959 17897 tablet_bootstrap.cc:492] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: Bootstrap starting.
I20260620 14:24:23.029601 17896 tablet_bootstrap.cc:654] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:23.030216 17897 tablet_bootstrap.cc:654] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:23.030472 17896 log.cc:824] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:23.031248 17898 tablet_bootstrap.cc:492] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480: Bootstrap starting.
I20260620 14:24:23.031381 17897 log.cc:824] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:23.032739 17896 tablet_bootstrap.cc:492] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: No bootstrap required, opened a new log
I20260620 14:24:23.032939 17896 ts_tablet_manager.cc:1397] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:24:23.033205 17897 tablet_bootstrap.cc:492] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: No bootstrap required, opened a new log
I20260620 14:24:23.033344 17897 ts_tablet_manager.cc:1397] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:24:23.035856 17896 raft_consensus.cc:348] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } }
I20260620 14:24:23.036124 17896 raft_consensus.cc:374] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:23.035964 17897 raft_consensus.cc:348] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } }
I20260620 14:24:23.036182 17896 raft_consensus.cc:729] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34f23077c2440028ca754af5d5400ff, State: Initialized, Role: FOLLOWER
I20260620 14:24:23.036190 17897 raft_consensus.cc:374] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:23.036249 17897 raft_consensus.cc:729] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e65c3b45a04613bfc1f623ea97eec4, State: Initialized, Role: FOLLOWER
I20260620 14:24:23.036684 17896 consensus_queue.cc:260] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } }
I20260620 14:24:23.036695 17897 consensus_queue.cc:260] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } }
I20260620 14:24:23.037322 17879 heartbeater.cc:503] Master 127.0.98.62:46841 was elected leader, sending a full tablet report...
I20260620 14:24:23.037322 17745 heartbeater.cc:503] Master 127.0.98.62:46841 was elected leader, sending a full tablet report...
I20260620 14:24:23.037689 17896 ts_tablet_manager.cc:1428] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:24:23.037690 17897 ts_tablet_manager.cc:1428] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260620 14:24:23.040117 17898 tablet_bootstrap.cc:654] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:23.041663 17898 log.cc:824] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:23.043478 17898 tablet_bootstrap.cc:492] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480: No bootstrap required, opened a new log
I20260620 14:24:23.043635 17898 ts_tablet_manager.cc:1397] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260620 14:24:23.046417 17898 raft_consensus.cc:348] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } }
I20260620 14:24:23.046733 17898 raft_consensus.cc:374] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:23.046797 17898 raft_consensus.cc:729] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08638bffaa743e9a924a7d491c15480, State: Initialized, Role: FOLLOWER
I20260620 14:24:23.047262 17898 consensus_queue.cc:260] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } }
I20260620 14:24:23.047983 17611 heartbeater.cc:503] Master 127.0.98.62:46841 was elected leader, sending a full tablet report...
I20260620 14:24:23.048251 17898 ts_tablet_manager.cc:1428] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 14:24:23.111951 17904 raft_consensus.cc:482] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:23.112227 17904 raft_consensus.cc:504] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } }
I20260620 14:24:23.113375 17904 leader_election.cc:288] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d34f23077c2440028ca754af5d5400ff (127.0.98.2:38265), c0e65c3b45a04613bfc1f623ea97eec4 (127.0.98.3:39179)
I20260620 14:24:23.117915 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.113783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35393 (local address 127.0.98.2:38265)
0620 14:24:23.113916 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:23.113921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:23.114279 (+ 358us) server_negotiation.cc:407] Connection header received
0620 14:24:23.114440 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:23.114446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:23.114514 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:23.114573 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:23.114977 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.115551 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.116822 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.117245 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.117387 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.117463 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:23.117700 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:24:23.117777 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:24:23.117959 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.113683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38265 (local address 127.0.98.1:35393)
0620 14:24:23.114135 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:23.114147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:23.114331 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:23.114616 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:23.114619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:23.114635 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:23.114834 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.114840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.115667 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.115671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.116678 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.116688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.117400 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.117403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.117497 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.117520 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:24:23.117684 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 14:24:23.117761 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":98,"spinlock_wait_cycles":55168,"thread_start_us":117,"threads_started":1}
I20260620 14:24:23.118039 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.113683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39179 (local address 127.0.98.1:41717)
0620 14:24:23.114156 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:23.114169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:23.114322 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:23.114660 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:23.114664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:23.114685 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:23.114828 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.114833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.115673 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.115676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.116884 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.116894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.117568 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.117572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.117686 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.117707 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:24:23.117806 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:24:23.117876 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":76,"spinlock_wait_cycles":36352,"thread_start_us":115,"threads_started":1}
I20260620 14:24:23.118083 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.113773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41717 (local address 127.0.98.3:39179)
0620 14:24:23.113916 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:23.113921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:23.114241 (+ 320us) server_negotiation.cc:407] Connection header received
0620 14:24:23.114482 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:23.114484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:23.114530 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:23.114626 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:23.114944 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.115555 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.117007 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.117445 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.117566 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.117631 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:23.117823 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:24:23.117895 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:24:23.118750 17833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd337546987437885f5287ba5543424" candidate_uuid: "d08638bffaa743e9a924a7d491c15480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" is_pre_election: true
I20260620 14:24:23.119027 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd337546987437885f5287ba5543424" candidate_uuid: "d08638bffaa743e9a924a7d491c15480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d34f23077c2440028ca754af5d5400ff" is_pre_election: true
I20260620 14:24:23.119134 17833 raft_consensus.cc:2454] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d08638bffaa743e9a924a7d491c15480 in term 0.
I20260620 14:24:23.119331 17699 raft_consensus.cc:2454] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d08638bffaa743e9a924a7d491c15480 in term 0.
I20260620 14:24:23.119639 17498 leader_election.cc:302] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0e65c3b45a04613bfc1f623ea97eec4, d08638bffaa743e9a924a7d491c15480; no voters:
I20260620 14:24:23.119966 17904 raft_consensus.cc:2781] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:23.120043 17904 raft_consensus.cc:482] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:23.120074 17904 raft_consensus.cc:3037] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:23.121111 17904 raft_consensus.cc:504] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } }
I20260620 14:24:23.121475 17904 leader_election.cc:288] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [CANDIDATE]: Term 1 election: Requested vote from peers d34f23077c2440028ca754af5d5400ff (127.0.98.2:38265), c0e65c3b45a04613bfc1f623ea97eec4 (127.0.98.3:39179)
I20260620 14:24:23.121872 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd337546987437885f5287ba5543424" candidate_uuid: "d08638bffaa743e9a924a7d491c15480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d34f23077c2440028ca754af5d5400ff"
I20260620 14:24:23.121872 17833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd337546987437885f5287ba5543424" candidate_uuid: "d08638bffaa743e9a924a7d491c15480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e65c3b45a04613bfc1f623ea97eec4"
I20260620 14:24:23.122000 17833 raft_consensus.cc:3037] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:23.122000 17699 raft_consensus.cc:3037] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:23.123317 17699 raft_consensus.cc:2454] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d08638bffaa743e9a924a7d491c15480 in term 1.
I20260620 14:24:23.123317 17833 raft_consensus.cc:2454] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d08638bffaa743e9a924a7d491c15480 in term 1.
I20260620 14:24:23.123651 17500 leader_election.cc:302] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d08638bffaa743e9a924a7d491c15480, d34f23077c2440028ca754af5d5400ff; no voters:
I20260620 14:24:23.123875 17904 raft_consensus.cc:2781] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:23.124145 17904 raft_consensus.cc:686] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [term 1 LEADER]: Becoming Leader. State: Replica: d08638bffaa743e9a924a7d491c15480, State: Running, Role: LEADER
I20260620 14:24:23.124475 17904 consensus_queue.cc:237] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } }
I20260620 14:24:23.127604 17418 catalog_manager.cc:5495] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 reported cstate change: term changed from 0 to 1, leader changed from <none> to d08638bffaa743e9a924a7d491c15480 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d08638bffaa743e9a924a7d491c15480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08638bffaa743e9a924a7d491c15480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:23.167460 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.164266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.0.98.1:46639)
0620 14:24:23.164399 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:23.164403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:23.164415 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:24:23.164496 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:23.164498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:23.164550 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:23.164647 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:23.164824 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.165331 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.166165 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.166383 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.166468 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.166715 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:23.167155 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:23.167245 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:23.167299 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:24:23.167349 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:24:23.173287 17699 raft_consensus.cc:1261] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [term 1 FOLLOWER]: Refusing update from remote peer d08638bffaa743e9a924a7d491c15480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:23.173287 17833 raft_consensus.cc:1261] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [term 1 FOLLOWER]: Refusing update from remote peer d08638bffaa743e9a924a7d491c15480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:23.174059 17904 consensus_queue.cc:1035] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e65c3b45a04613bfc1f623ea97eec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:23.174297 17907 consensus_queue.cc:1035] T 8bd337546987437885f5287ba5543424 P d08638bffaa743e9a924a7d491c15480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d34f23077c2440028ca754af5d5400ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:23.182258 17910 mvcc.cc:205] Tried to move back new op lower bound from 7298930537150492672 to 7298930536960409600. Current Snapshot: MvccSnapshot[applied={T|T < 7298930537150492672}]
I20260620 14:24:23.185492 17911 mvcc.cc:205] Tried to move back new op lower bound from 7298930537150492672 to 7298930536960409600. Current Snapshot: MvccSnapshot[applied={T|T < 7298930537150492672}]
I20260620 14:24:23.191362 17418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:23.191517 17418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:23.194046 17418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca707caad7604acc96ad693fe9afdb3b: Sending DeleteTablet for 3 replicas of tablet 8bd337546987437885f5287ba5543424
I20260620 14:24:23.194936 17545 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd337546987437885f5287ba5543424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42804
I20260620 14:24:23.194936 17813 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd337546987437885f5287ba5543424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33366
I20260620 14:24:23.194907 17679 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd337546987437885f5287ba5543424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48206
I20260620 14:24:23.195473 17920 tablet_replica.cc:330] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: stopping tablet replica
I20260620 14:24:23.195469 17922 tablet_replica.cc:330] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: stopping tablet replica
I20260620 14:24:23.195832 17922 raft_consensus.cc:2229] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:23.195858 17920 raft_consensus.cc:2229] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:23.196161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 17482
I20260620 14:24:23.196162 17920 raft_consensus.cc:2258] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:23.196749 17922 raft_consensus.cc:2258] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:23.198081 17920 ts_tablet_manager.cc:1905] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:23.198242 17922 ts_tablet_manager.cc:1905] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:23.200737 17922 ts_tablet_manager.cc:1918] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:23.200876 17922 log.cc:1196] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/8bd337546987437885f5287ba5543424
I20260620 14:24:23.201272 17922 ts_tablet_manager.cc:1939] T 8bd337546987437885f5287ba5543424 P c0e65c3b45a04613bfc1f623ea97eec4: Deleting consensus metadata
I20260620 14:24:23.201462 17920 ts_tablet_manager.cc:1918] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:23.201561 17920 log.cc:1196] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/8bd337546987437885f5287ba5543424
I20260620 14:24:23.201882 17920 ts_tablet_manager.cc:1939] T 8bd337546987437885f5287ba5543424 P d34f23077c2440028ca754af5d5400ff: Deleting consensus metadata
I20260620 14:24:23.202610 17404 catalog_manager.cc:4841] TS c0e65c3b45a04613bfc1f623ea97eec4 (127.0.98.3:39179): tablet 8bd337546987437885f5287ba5543424 (table test-table [id=27cad5cbc399469897c9f00e781ffb19]) successfully deleted
I20260620 14:24:23.203044 17406 catalog_manager.cc:4841] TS d34f23077c2440028ca754af5d5400ff (127.0.98.2:38265): tablet 8bd337546987437885f5287ba5543424 (table test-table [id=27cad5cbc399469897c9f00e781ffb19]) successfully deleted
I20260620 14:24:23.204581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 17616
W20260620 14:24:23.204582 17404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46639 (error 108)
I20260620 14:24:23.206235 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.205860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46639 (local address 127.0.0.1:42816)
0620 14:24:23.206057 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:23.206133 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 14:24:23.206409 17404 catalog_manager.cc:4584] TS d08638bffaa743e9a924a7d491c15480 (127.0.98.1:46639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bd337546987437885f5287ba5543424: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46639: connect: Connection refused (error 111)
I20260620 14:24:23.210052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 17750
I20260620 14:24:23.215559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 17388
2026-06-20T14:24:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2308 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:24:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:23Z Disabled control of system clock
I20260620 14:24:23.245939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33729
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:24:23.355906 17927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:23.356169 17927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:23.356204 17927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:23.360147 17927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:23.360208 17927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:23.360224 17927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:23.360241 17927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:23.360256 17927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:23.365051 17927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:23.366072 17927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:23.367406 17927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:23.375881 17935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:23.375881 17932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:23.376178 17927 server_base.cc:1034] running on GCE node
W20260620 14:24:23.375933 17933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:23.376796 17927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:23.377918 17927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:23.379109 17927 hybrid_clock.cc:648] HybridClock initialized: now 1781965463379093 us; error 28 us; skew 500 ppm
I20260620 14:24:23.381007 17927 webserver.cc:460] Webserver started at http://127.0.98.62:46235/ using document root <none> and password file <none>
I20260620 14:24:23.381599 17927 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:23.381692 17927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:23.381933 17927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:23.384066 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "d002fb9c77504c2ca44dd11e7e8db28e"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "32d9d712694c28cfa2e379b47ac490f1"
server_key_iv: "1bd3e80bc92b62d69587df510a9947bd"
server_key_version: "encryptionkey@0"
I20260620 14:24:23.384624 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "d002fb9c77504c2ca44dd11e7e8db28e"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "32d9d712694c28cfa2e379b47ac490f1"
server_key_iv: "1bd3e80bc92b62d69587df510a9947bd"
server_key_version: "encryptionkey@0"
I20260620 14:24:23.388250 17927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:23.390551 17941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:23.391736 17927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:23.391920 17927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "d002fb9c77504c2ca44dd11e7e8db28e"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "32d9d712694c28cfa2e379b47ac490f1"
server_key_iv: "1bd3e80bc92b62d69587df510a9947bd"
server_key_version: "encryptionkey@0"
I20260620 14:24:23.392037 17927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:23.406041 17927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:23.406711 17927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:23.406906 17927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:23.413877 17927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33729
I20260620 14:24:23.413884 17993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33729 every 8 connection(s)
I20260620 14:24:23.414845 17927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:23.417896 17994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:23.421669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 17927
I20260620 14:24:23.421797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:23.422103 392 external_mini_cluster.cc:1428] Setting key 18f3fd38436602e588c9539e50eebadb
I20260620 14:24:23.423851 17994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e: Bootstrap starting.
I20260620 14:24:23.425977 17994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:23.426687 17994 log.cc:824] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:23.428503 17994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e: No bootstrap required, opened a new log
I20260620 14:24:23.431217 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.423519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.0.98.62:33729)
0620 14:24:23.424230 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:23.424245 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:23.424280 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:24:23.424904 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:23.424922 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:23.425305 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:23.425644 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:23.426106 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.427158 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.428096 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.428588 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.428764 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.429613 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:23.429618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:23.429630 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:23.429995 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:23.430046 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:23.430064 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:23.430172 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:23.430315 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:24:23.430560 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260620 14:24:23.432143 17994 raft_consensus.cc:348] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002fb9c77504c2ca44dd11e7e8db28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33729 } }
I20260620 14:24:23.432346 17994 raft_consensus.cc:374] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:23.432430 17994 raft_consensus.cc:729] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d002fb9c77504c2ca44dd11e7e8db28e, State: Initialized, Role: FOLLOWER
I20260620 14:24:23.432900 17994 consensus_queue.cc:260] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002fb9c77504c2ca44dd11e7e8db28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33729 } }
I20260620 14:24:23.433027 17994 raft_consensus.cc:388] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:23.433109 17994 raft_consensus.cc:482] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:23.433233 17994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:23.434221 17994 raft_consensus.cc:504] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002fb9c77504c2ca44dd11e7e8db28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33729 } }
I20260620 14:24:23.434518 17994 leader_election.cc:302] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d002fb9c77504c2ca44dd11e7e8db28e; no voters:
I20260620 14:24:23.434756 17994 leader_election.cc:288] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:23.434917 17999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:23.435149 17999 raft_consensus.cc:686] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [term 1 LEADER]: Becoming Leader. State: Replica: d002fb9c77504c2ca44dd11e7e8db28e, State: Running, Role: LEADER
I20260620 14:24:23.435459 17999 consensus_queue.cc:237] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002fb9c77504c2ca44dd11e7e8db28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33729 } }
I20260620 14:24:23.436010 17994 sys_catalog.cc:564] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:23.437057 18001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d002fb9c77504c2ca44dd11e7e8db28e. Latest consensus state: current_term: 1 leader_uuid: "d002fb9c77504c2ca44dd11e7e8db28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002fb9c77504c2ca44dd11e7e8db28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33729 } } }
I20260620 14:24:23.437191 18001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:23.437453 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d002fb9c77504c2ca44dd11e7e8db28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002fb9c77504c2ca44dd11e7e8db28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33729 } } }
I20260620 14:24:23.437541 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:23.437793 18008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:23.440872 18008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:23.446522 18008 catalog_manager.cc:1347] Generated new cluster ID: 035aeffc8fb64b27a085ab4e86fc0b15
I20260620 14:24:23.446595 18008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:23.457152 18008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:23.458118 18008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:23.464075 18008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e: Generated new TSK 0
I20260620 14:24:23.464929 18008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:23.475997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33729
--builtin_ntp_servers=127.0.98.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:24:23.595481 18018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:23.595750 18018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:23.595872 18018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:23.599989 18018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:23.600042 18018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:23.600097 18018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:23.604772 18018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:23.605820 18018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:23.607187 18018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:23.616712 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:23.616712 18024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:23.616768 18026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:23.616940 18018 server_base.cc:1034] running on GCE node
I20260620 14:24:23.617470 18018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:23.618091 18018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:23.619292 18018 hybrid_clock.cc:648] HybridClock initialized: now 1781965463619274 us; error 30 us; skew 500 ppm
I20260620 14:24:23.621181 18018 webserver.cc:460] Webserver started at http://127.0.98.1:39431/ using document root <none> and password file <none>
I20260620 14:24:23.621760 18018 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:23.621853 18018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:23.622081 18018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:23.624136 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "c77bfcffd21c486ebde11def7d955dd2"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "2bad048dcae593d71c87789ac0844c95"
server_key_iv: "023025882cadcef88d7bdb06aeb36181"
server_key_version: "encryptionkey@0"
I20260620 14:24:23.624768 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c77bfcffd21c486ebde11def7d955dd2"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "2bad048dcae593d71c87789ac0844c95"
server_key_iv: "023025882cadcef88d7bdb06aeb36181"
server_key_version: "encryptionkey@0"
I20260620 14:24:23.628342 18018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:23.630754 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:23.632000 18018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:23.632125 18018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "c77bfcffd21c486ebde11def7d955dd2"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "2bad048dcae593d71c87789ac0844c95"
server_key_iv: "023025882cadcef88d7bdb06aeb36181"
server_key_version: "encryptionkey@0"
I20260620 14:24:23.632236 18018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:23.653177 18018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:23.653920 18018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:23.654129 18018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:23.654768 18018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:23.655721 18018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:23.655821 18018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:23.655907 18018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:23.655957 18018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:23.664919 18018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46039
I20260620 14:24:23.664945 18145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46039 every 8 connection(s)
I20260620 14:24:23.665804 18018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:23.672935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 18018
I20260620 14:24:23.673066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:23.673331 392 external_mini_cluster.cc:1428] Setting key 01872ea7e0cfb9fd36ad52b0eaae66bf
I20260620 14:24:23.675223 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.667536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51499 (local address 127.0.98.62:33729)
0620 14:24:23.667674 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:23.667678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:23.668762 (+ 1084us) server_negotiation.cc:407] Connection header received
0620 14:24:23.670198 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:23.670202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:23.670260 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:23.670352 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:23.671255 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.671836 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.672998 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.673320 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.673434 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.674661 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:23.674664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:23.674667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:23.674679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:23.674704 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:23.674716 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:23.674781 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:23.675022 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:24:23.675084 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:24:23.675491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33729
--builtin_ntp_servers=127.0.98.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:24:23.676148 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.667950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33729 (local address 127.0.98.1:51499)
0620 14:24:23.668594 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:23.668637 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:23.669978 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:23.670520 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:23.670532 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:23.670615 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:23.671104 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.671116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.671968 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.671973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.672860 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.672869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.673486 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.673490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.673648 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.674342 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:23.674344 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:23.674395 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:23.674468 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:23.674846 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:23.674858 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:23.674865 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:24:23.674993 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:24:23.675210 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260620 14:24:23.677279 18146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33729
I20260620 14:24:23.677505 18146 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:23.678018 18146 heartbeater.cc:511] Master 127.0.98.62:33729 requested a full tablet report, sending...
I20260620 14:24:23.679589 17958 ts_manager.cc:194] Registered new tserver with Master: c77bfcffd21c486ebde11def7d955dd2 (127.0.98.1:46039)
I20260620 14:24:23.681370 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51499
W20260620 14:24:23.787119 18149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:23.787361 18149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:23.787397 18149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:23.792204 18149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:23.792271 18149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:23.792364 18149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:23.797462 18149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:23.798518 18149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:23.799964 18149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:23.809320 18154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:23.809355 18157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:23.809424 18155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:23.809767 18149 server_base.cc:1034] running on GCE node
I20260620 14:24:23.810237 18149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:23.810858 18149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:23.812047 18149 hybrid_clock.cc:648] HybridClock initialized: now 1781965463812032 us; error 34 us; skew 500 ppm
I20260620 14:24:23.813948 18149 webserver.cc:460] Webserver started at http://127.0.98.2:40685/ using document root <none> and password file <none>
I20260620 14:24:23.814518 18149 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:23.814608 18149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:23.814810 18149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:23.816869 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "90901a37e4bc4c199d1cd7a5094f8660"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "fd5b8cd9f4a1c35a86300d1fa1f8a789"
server_key_iv: "0ed1e38974d2fe8e1748fc464d21d2c1"
server_key_version: "encryptionkey@0"
I20260620 14:24:23.817435 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "90901a37e4bc4c199d1cd7a5094f8660"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "fd5b8cd9f4a1c35a86300d1fa1f8a789"
server_key_iv: "0ed1e38974d2fe8e1748fc464d21d2c1"
server_key_version: "encryptionkey@0"
I20260620 14:24:23.820974 18149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:23.823468 18163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:23.824756 18149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:23.824891 18149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "90901a37e4bc4c199d1cd7a5094f8660"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "fd5b8cd9f4a1c35a86300d1fa1f8a789"
server_key_iv: "0ed1e38974d2fe8e1748fc464d21d2c1"
server_key_version: "encryptionkey@0"
I20260620 14:24:23.825003 18149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:23.846181 18149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:23.846958 18149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:23.847167 18149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:23.847811 18149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:23.848783 18149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:23.848853 18149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:23.848918 18149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:23.848960 18149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:23.858103 18149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44889
I20260620 14:24:23.858130 18276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44889 every 8 connection(s)
I20260620 14:24:23.858999 18149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:23.862820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 18149
I20260620 14:24:23.862916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:23.863197 392 external_mini_cluster.cc:1428] Setting key d771a6f3de8be970ac1a27358bd28da3
I20260620 14:24:23.865209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33729
--builtin_ntp_servers=127.0.98.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:24:23.868788 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.860823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36521 (local address 127.0.98.62:33729)
0620 14:24:23.861021 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:23.861025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:23.862030 (+ 1005us) server_negotiation.cc:407] Connection header received
0620 14:24:23.863527 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:23.863530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:23.863588 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:23.863666 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:23.864777 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.865299 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.866646 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.866866 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.866957 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.868316 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:23.868319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:23.868321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:23.868332 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:23.868357 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:23.868366 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:23.868418 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:23.868609 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 14:24:23.868655 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 14:24:23.870026 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:23.861239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33729 (local address 127.0.98.2:36521)
0620 14:24:23.861865 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:23.861914 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:23.863323 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:23.863881 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:23.863888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:23.864014 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:23.864629 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.864645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.865485 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.865488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.866501 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:23.866515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:23.867033 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:23.867039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:23.867257 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:23.868011 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:23.868016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:23.868071 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:23.868154 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:23.868450 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:23.868462 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:23.868471 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:24:23.868608 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:24:23.868907 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260620 14:24:23.871379 18277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33729
I20260620 14:24:23.871630 18277 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:23.872164 18277 heartbeater.cc:511] Master 127.0.98.62:33729 requested a full tablet report, sending...
I20260620 14:24:23.873189 17958 ts_manager.cc:194] Registered new tserver with Master: 90901a37e4bc4c199d1cd7a5094f8660 (127.0.98.2:44889)
I20260620 14:24:23.874046 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36521
W20260620 14:24:23.975961 18280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:23.976200 18280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:23.976238 18280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:23.980201 18280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:23.980255 18280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:23.980311 18280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:23.985090 18280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:23.986052 18280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:23.987409 18280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:23.996879 18288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:23.996896 18285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:23.997080 18280 server_base.cc:1034] running on GCE node
W20260620 14:24:23.996896 18286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:23.997551 18280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:23.998150 18280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:23.999323 18280 hybrid_clock.cc:648] HybridClock initialized: now 1781965463999309 us; error 27 us; skew 500 ppm
I20260620 14:24:24.001130 18280 webserver.cc:460] Webserver started at http://127.0.98.3:33323/ using document root <none> and password file <none>
I20260620 14:24:24.001639 18280 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:24.001695 18280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:24.001847 18280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:24.003757 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "654ba822966e484399dd883677db1b76"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "dce82011e9959ccde51193a8724fdf6f"
server_key_iv: "900b212c923b871a27f6dda32dc86a12"
server_key_version: "encryptionkey@0"
I20260620 14:24:24.004357 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "654ba822966e484399dd883677db1b76"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "dce82011e9959ccde51193a8724fdf6f"
server_key_iv: "900b212c923b871a27f6dda32dc86a12"
server_key_version: "encryptionkey@0"
I20260620 14:24:24.007874 18280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:24:24.010099 18294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:24.011242 18280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:24.011370 18280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "654ba822966e484399dd883677db1b76"
format_stamp: "Formatted at 2026-06-20 14:24:23 on dist-test-slave-87nc"
server_key: "dce82011e9959ccde51193a8724fdf6f"
server_key_iv: "900b212c923b871a27f6dda32dc86a12"
server_key_version: "encryptionkey@0"
I20260620 14:24:24.011488 18280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:24.036953 18280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:24.037752 18280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:24.037950 18280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:24.038561 18280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:24.039534 18280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:24.039604 18280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:24.039676 18280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:24.039721 18280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:24.048175 18280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36695
I20260620 14:24:24.048202 18407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36695 every 8 connection(s)
I20260620 14:24:24.049080 18280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:24.051748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 18280
I20260620 14:24:24.051870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:24.052152 392 external_mini_cluster.cc:1428] Setting key f6c20a3bc3bfb6e7cf3bb9825865f545
I20260620 14:24:24.058386 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.050892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36405 (local address 127.0.98.62:33729)
0620 14:24:24.051039 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.051043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.052229 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 14:24:24.053600 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.053603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.053658 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.053763 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:24.054721 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.055253 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.056460 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.056687 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.056798 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.057901 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:24.057905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:24.057908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:24.057928 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:24.057948 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:24.057956 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:24.058031 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.058187 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.058247 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:24:24.059234 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.051312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33729 (local address 127.0.98.3:36405)
0620 14:24:24.052078 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.052120 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.053391 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.053936 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.053943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.054058 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:24.054589 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.054602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.055372 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.055376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.056331 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.056340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.056832 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.056836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.057006 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.057643 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:24.057644 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:24.057685 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:24.057748 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:24.058045 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:24.058057 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:24.058066 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.058187 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.058466 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
I20260620 14:24:24.060330 18408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33729
I20260620 14:24:24.060551 18408 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:24.061023 18408 heartbeater.cc:511] Master 127.0.98.62:33729 requested a full tablet report, sending...
I20260620 14:24:24.061998 17958 ts_manager.cc:194] Registered new tserver with Master: 654ba822966e484399dd883677db1b76 (127.0.98.3:36695)
I20260620 14:24:24.062517 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36405
I20260620 14:24:24.065745 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:24.071013 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.067561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.0.98.62:33729)
0620 14:24:24.067685 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.067707 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.067980 (+ 273us) server_negotiation.cc:407] Connection header received
0620 14:24:24.068078 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.068080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.068118 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.068184 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:24.068412 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.068893 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.069657 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.069856 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.069946 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.070667 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:24.070669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:24.070671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:24.070681 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:24.070696 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:24.070704 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:24.070770 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.070857 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.070917 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:24:24.073668 17958 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:24.076694 17958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:24.090659 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.085182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.0.98.1:46039)
0620 14:24:24.086077 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.086088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.086112 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:24:24.086188 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.086194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.086385 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.086547 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:24.086969 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.088054 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.089473 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.090009 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.090129 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.090322 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.090407 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.090503 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":111,"threads_started":1}
I20260620 14:24:24.090673 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.085303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36695 (local address 127.0.0.1:37198)
0620 14:24:24.085812 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.085825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.085928 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.086542 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.086547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.086568 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:24.086670 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.086679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.087669 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.087675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.089103 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.089112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.090198 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.090205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.090296 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.090318 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.090365 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.090497 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":123,"thread_start_us":108,"threads_started":1}
I20260620 14:24:24.090655 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.084950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46039 (local address 127.0.0.1:44278)
0620 14:24:24.085306 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.085357 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.085491 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.086707 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.086710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.086729 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:24.086831 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.086837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.088229 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.088233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.089350 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.089359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.090129 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.090133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.090235 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.090270 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.090379 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.090488 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":10,"thread_start_us":97,"threads_started":1}
I20260620 14:24:24.091163 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.084774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.0.98.2:44889)
0620 14:24:24.085653 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.085666 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.085883 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:24:24.085971 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.085977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.086172 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.086329 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:24.086621 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.087834 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.089403 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.090168 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.090259 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.090439 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.090903 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.091008 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":144,"threads_started":1}
I20260620 14:24:24.091449 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.084586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44889 (local address 127.0.0.1:52556)
0620 14:24:24.085118 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.085707 (+ 589us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.085787 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.086329 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.086335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.086356 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:24.086485 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.086495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.087950 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.087953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.089276 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.089287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.090497 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.090500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.090582 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.090602 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.090668 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.091325 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":56,"thread_start_us":103,"threads_started":1}
I20260620 14:24:24.091467 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.085753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.0.98.3:36695)
0620 14:24:24.086173 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.086183 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.086201 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:24:24.086259 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.086264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.086400 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.086547 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:24.086762 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.087553 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.089227 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.090053 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.090740 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.090939 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.090982 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.091061 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":174,"threads_started":1}
I20260620 14:24:24.093886 18080 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c60d7e264cf1a9245df1373f58ca (DEFAULT_TABLE table=test-table [id=5e215df3a82b4f85bfb4967cdfce8eab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:24.094039 18342 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c60d7e264cf1a9245df1373f58ca (DEFAULT_TABLE table=test-table [id=5e215df3a82b4f85bfb4967cdfce8eab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:24.094306 18211 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c60d7e264cf1a9245df1373f58ca (DEFAULT_TABLE table=test-table [id=5e215df3a82b4f85bfb4967cdfce8eab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:24.094753 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c60d7e264cf1a9245df1373f58ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:24.095016 18342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c60d7e264cf1a9245df1373f58ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:24.095122 18211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c60d7e264cf1a9245df1373f58ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:24.101339 18425 tablet_bootstrap.cc:492] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2: Bootstrap starting.
I20260620 14:24:24.102909 18427 tablet_bootstrap.cc:492] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660: Bootstrap starting.
I20260620 14:24:24.103480 18425 tablet_bootstrap.cc:654] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:24.104111 18426 tablet_bootstrap.cc:492] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: Bootstrap starting.
I20260620 14:24:24.104326 18425 log.cc:824] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:24.104874 18427 tablet_bootstrap.cc:654] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:24.105688 18427 log.cc:824] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:24.106184 18426 tablet_bootstrap.cc:654] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:24.106766 18425 tablet_bootstrap.cc:492] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2: No bootstrap required, opened a new log
I20260620 14:24:24.106930 18426 log.cc:824] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:24.106963 18425 ts_tablet_manager.cc:1397] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:24:24.107728 18427 tablet_bootstrap.cc:492] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660: No bootstrap required, opened a new log
I20260620 14:24:24.107949 18427 ts_tablet_manager.cc:1397] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 14:24:24.108557 18426 tablet_bootstrap.cc:492] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: No bootstrap required, opened a new log
I20260620 14:24:24.108747 18426 ts_tablet_manager.cc:1397] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:24:24.111299 18425 raft_consensus.cc:348] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } }
I20260620 14:24:24.111577 18425 raft_consensus.cc:374] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:24.111653 18425 raft_consensus.cc:729] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77bfcffd21c486ebde11def7d955dd2, State: Initialized, Role: FOLLOWER
I20260620 14:24:24.111774 18426 raft_consensus.cc:348] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } }
I20260620 14:24:24.112067 18426 raft_consensus.cc:374] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:24.112121 18426 raft_consensus.cc:729] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654ba822966e484399dd883677db1b76, State: Initialized, Role: FOLLOWER
I20260620 14:24:24.111999 18427 raft_consensus.cc:348] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } }
I20260620 14:24:24.112108 18425 consensus_queue.cc:260] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } }
I20260620 14:24:24.112535 18426 consensus_queue.cc:260] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } }
I20260620 14:24:24.112737 18146 heartbeater.cc:503] Master 127.0.98.62:33729 was elected leader, sending a full tablet report...
I20260620 14:24:24.113037 18427 raft_consensus.cc:374] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:24.113098 18427 raft_consensus.cc:729] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90901a37e4bc4c199d1cd7a5094f8660, State: Initialized, Role: FOLLOWER
I20260620 14:24:24.113106 18425 ts_tablet_manager.cc:1428] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:24:24.113484 18426 ts_tablet_manager.cc:1428] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:24:24.113521 18427 consensus_queue.cc:260] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } }
I20260620 14:24:24.113816 18408 heartbeater.cc:503] Master 127.0.98.62:33729 was elected leader, sending a full tablet report...
I20260620 14:24:24.114634 18427 ts_tablet_manager.cc:1428] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260620 14:24:24.115754 18277 heartbeater.cc:503] Master 127.0.98.62:33729 was elected leader, sending a full tablet report...
I20260620 14:24:24.165951 18433 raft_consensus.cc:482] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:24.166209 18433 raft_consensus.cc:504] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } }
W20260620 14:24:24.166931 18147 tablet.cc:2369] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:24.167416 18433 leader_election.cc:288] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 654ba822966e484399dd883677db1b76 (127.0.98.3:36695), c77bfcffd21c486ebde11def7d955dd2 (127.0.98.1:46039)
I20260620 14:24:24.171227 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.167893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54663 (local address 127.0.98.1:46039)
0620 14:24:24.168081 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.168085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.168212 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:24:24.168307 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.168310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.168360 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.168454 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:24.168732 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.169239 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.170272 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.170650 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.170779 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.170843 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.171006 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.171093 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 14:24:24.171236 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.167647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36695 (local address 127.0.98.2:43087)
0620 14:24:24.167793 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.167810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.167920 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.168219 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.168224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.168246 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:24.168370 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.168379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.169196 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.169200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.170061 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.170070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.170738 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.170745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.170845 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.170874 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.170998 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.171124 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":25}
I20260620 14:24:24.171233 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.167800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43087 (local address 127.0.98.3:36695)
0620 14:24:24.167940 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.167947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.167969 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:24:24.168031 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.168034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.168082 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.168175 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:24.168500 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.169088 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.170187 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.170616 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.170696 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.170778 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.171025 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.171105 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:24:24.171252 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.167655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46039 (local address 127.0.98.2:54663)
0620 14:24:24.168120 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.168135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.168226 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.168494 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.168497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.168513 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:24.168622 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.168627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.169350 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.169353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.170152 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.170163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.170807 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.170811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.170888 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.170904 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.170984 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.171110 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":119,"thread_start_us":119,"threads_started":1}
I20260620 14:24:24.171943 18362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c60d7e264cf1a9245df1373f58ca" candidate_uuid: "90901a37e4bc4c199d1cd7a5094f8660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654ba822966e484399dd883677db1b76" is_pre_election: true
I20260620 14:24:24.171937 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c60d7e264cf1a9245df1373f58ca" candidate_uuid: "90901a37e4bc4c199d1cd7a5094f8660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77bfcffd21c486ebde11def7d955dd2" is_pre_election: true
I20260620 14:24:24.172233 18100 raft_consensus.cc:2454] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90901a37e4bc4c199d1cd7a5094f8660 in term 0.
I20260620 14:24:24.172233 18362 raft_consensus.cc:2454] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90901a37e4bc4c199d1cd7a5094f8660 in term 0.
I20260620 14:24:24.172693 18165 leader_election.cc:302] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90901a37e4bc4c199d1cd7a5094f8660, c77bfcffd21c486ebde11def7d955dd2; no voters:
I20260620 14:24:24.172988 18433 raft_consensus.cc:2781] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:24.173074 18433 raft_consensus.cc:482] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:24.173116 18433 raft_consensus.cc:3037] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:24.174122 18433 raft_consensus.cc:504] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } }
I20260620 14:24:24.174513 18433 leader_election.cc:288] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [CANDIDATE]: Term 1 election: Requested vote from peers 654ba822966e484399dd883677db1b76 (127.0.98.3:36695), c77bfcffd21c486ebde11def7d955dd2 (127.0.98.1:46039)
I20260620 14:24:24.174817 18362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c60d7e264cf1a9245df1373f58ca" candidate_uuid: "90901a37e4bc4c199d1cd7a5094f8660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654ba822966e484399dd883677db1b76"
I20260620 14:24:24.174858 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c60d7e264cf1a9245df1373f58ca" candidate_uuid: "90901a37e4bc4c199d1cd7a5094f8660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77bfcffd21c486ebde11def7d955dd2"
I20260620 14:24:24.174952 18362 raft_consensus.cc:3037] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:24.174997 18100 raft_consensus.cc:3037] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:24.176049 18100 raft_consensus.cc:2454] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90901a37e4bc4c199d1cd7a5094f8660 in term 1.
I20260620 14:24:24.176060 18362 raft_consensus.cc:2454] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90901a37e4bc4c199d1cd7a5094f8660 in term 1.
I20260620 14:24:24.176337 18165 leader_election.cc:302] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90901a37e4bc4c199d1cd7a5094f8660, c77bfcffd21c486ebde11def7d955dd2; no voters:
I20260620 14:24:24.176525 18433 raft_consensus.cc:2781] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:24.176751 18433 raft_consensus.cc:686] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 1 LEADER]: Becoming Leader. State: Replica: 90901a37e4bc4c199d1cd7a5094f8660, State: Running, Role: LEADER
I20260620 14:24:24.177073 18433 consensus_queue.cc:237] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } }
I20260620 14:24:24.180222 17958 catalog_manager.cc:5495] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90901a37e4bc4c199d1cd7a5094f8660 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "90901a37e4bc4c199d1cd7a5094f8660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90901a37e4bc4c199d1cd7a5094f8660" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:24.211536 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.208372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.0.98.2:44889)
0620 14:24:24.208508 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.208512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.208525 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:24:24.208630 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.208632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.208685 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.208771 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:24.208976 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.209461 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.210243 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.210444 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.210516 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.210768 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:24.211164 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:24.211249 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.211292 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.211334 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:24:24.217250 18100 raft_consensus.cc:1261] T 5028c60d7e264cf1a9245df1373f58ca P c77bfcffd21c486ebde11def7d955dd2 [term 1 FOLLOWER]: Refusing update from remote peer 90901a37e4bc4c199d1cd7a5094f8660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:24.217250 18362 raft_consensus.cc:1261] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [term 1 FOLLOWER]: Refusing update from remote peer 90901a37e4bc4c199d1cd7a5094f8660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:24.217974 18435 consensus_queue.cc:1035] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77bfcffd21c486ebde11def7d955dd2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:24.218164 18433 consensus_queue.cc:1035] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654ba822966e484399dd883677db1b76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:24.228717 18438 mvcc.cc:205] Tried to move back new op lower bound from 7298930541426810880 to 7298930541271814144. Current Snapshot: MvccSnapshot[applied={T|T < 7298930541426810880 or (T in {7298930541426810880})}]
I20260620 14:24:24.233244 17956 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:24.233387 17956 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:24.235882 17956 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d002fb9c77504c2ca44dd11e7e8db28e: Sending DeleteTablet for 3 replicas of tablet 5028c60d7e264cf1a9245df1373f58ca
I20260620 14:24:24.236745 18342 tablet_service.cc:1514] Processing DeleteTablet for tablet 5028c60d7e264cf1a9245df1373f58ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:24 UTC) from {username='slave'} at 127.0.0.1:37198
I20260620 14:24:24.236748 18080 tablet_service.cc:1514] Processing DeleteTablet for tablet 5028c60d7e264cf1a9245df1373f58ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:24 UTC) from {username='slave'} at 127.0.0.1:44278
I20260620 14:24:24.237342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 18018
I20260620 14:24:24.237445 18211 tablet_service.cc:1514] Processing DeleteTablet for tablet 5028c60d7e264cf1a9245df1373f58ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:24 UTC) from {username='slave'} at 127.0.0.1:52556
I20260620 14:24:24.238149 18448 tablet_replica.cc:330] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: stopping tablet replica
I20260620 14:24:24.238550 18448 raft_consensus.cc:2229] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:24.238867 18448 raft_consensus.cc:2258] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:24.240540 18448 ts_tablet_manager.cc:1905] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:24.242784 18448 ts_tablet_manager.cc:1918] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:24.242905 18448 log.cc:1196] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/5028c60d7e264cf1a9245df1373f58ca
I20260620 14:24:24.243187 18448 ts_tablet_manager.cc:1939] T 5028c60d7e264cf1a9245df1373f58ca P 654ba822966e484399dd883677db1b76: Deleting consensus metadata
I20260620 14:24:24.244026 18449 tablet_replica.cc:330] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660: stopping tablet replica
I20260620 14:24:24.244431 18449 raft_consensus.cc:2229] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:24.244494 17942 catalog_manager.cc:4841] TS 654ba822966e484399dd883677db1b76 (127.0.98.3:36695): tablet 5028c60d7e264cf1a9245df1373f58ca (table test-table [id=5e215df3a82b4f85bfb4967cdfce8eab]) successfully deleted
I20260620 14:24:24.244890 18449 raft_consensus.cc:2258] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:24.246230 18449 ts_tablet_manager.cc:1905] T 5028c60d7e264cf1a9245df1373f58ca P 90901a37e4bc4c199d1cd7a5094f8660: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:24:24.246513 17943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46039 (error 108)
I20260620 14:24:24.247335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 18149
I20260620 14:24:24.247676 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.247312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46039 (local address 127.0.0.1:44284)
0620 14:24:24.247514 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.247587 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260620 14:24:24.248116 17943 catalog_manager.cc:4584] TS c77bfcffd21c486ebde11def7d955dd2 (127.0.98.1:46039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5028c60d7e264cf1a9245df1373f58ca: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46039: connect: Connection refused (error 111)
I20260620 14:24:24.252969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 18280
I20260620 14:24:24.254166 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.253798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44889 (local address 127.0.0.1:52578)
0620 14:24:24.254001 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.254062 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 14:24:24.258778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 17927
2026-06-20T14:24:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1042 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:24 dist-test-slave-87nc krb5kdc[18453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:24 dist-test-slave-87nc krb5kdc[18453](info): set up 2 sockets
Jun 20 14:24:24 dist-test-slave-87nc krb5kdc[18453](info): commencing operation
krb5kdc: starting...
W20260620 14:24:25.295604 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:25 dist-test-slave-87nc krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:25.416785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:25.534180 18466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:25.534514 18466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:25.534595 18466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:25.538825 18466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:25.538923 18466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:25.538961 18466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:25.538995 18466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:25.539023 18466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:25.544482 18466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:25.545589 18466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:25.546666 18466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:25.555589 18471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:25.555718 18466 server_base.cc:1034] running on GCE node
W20260620 14:24:25.555606 18474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:25.555606 18472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:25.556519 18466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:25.557611 18466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:25.558825 18466 hybrid_clock.cc:648] HybridClock initialized: now 1781965465558801 us; error 40 us; skew 500 ppm
Jun 20 14:24:25 dist-test-slave-87nc krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:25.561754 18466 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:25.562755 18466 webserver.cc:460] Webserver started at http://127.0.98.62:36735/ using document root <none> and password file <none>
I20260620 14:24:25.563407 18466 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:25.563504 18466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:25.563741 18466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:25.565893 18466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "76fd4370a39b4c46a149a2073f794feb"
format_stamp: "Formatted at 2026-06-20 14:24:25 on dist-test-slave-87nc"
server_key: "b8aca75320324ed7b1152aba2f7a7019"
server_key_iv: "769dd625242d1d16e4a2d871d237b307"
server_key_version: "encryptionkey@0"
I20260620 14:24:25.566480 18466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "76fd4370a39b4c46a149a2073f794feb"
format_stamp: "Formatted at 2026-06-20 14:24:25 on dist-test-slave-87nc"
server_key: "b8aca75320324ed7b1152aba2f7a7019"
server_key_iv: "769dd625242d1d16e4a2d871d237b307"
server_key_version: "encryptionkey@0"
I20260620 14:24:25.570391 18466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:24:25.572865 18481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:25.574120 18466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:25.574273 18466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "76fd4370a39b4c46a149a2073f794feb"
format_stamp: "Formatted at 2026-06-20 14:24:25 on dist-test-slave-87nc"
server_key: "b8aca75320324ed7b1152aba2f7a7019"
server_key_iv: "769dd625242d1d16e4a2d871d237b307"
server_key_version: "encryptionkey@0"
I20260620 14:24:25.574394 18466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:25.591583 18466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:25.595122 18466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:25.595340 18466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:25.602607 18466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45613
I20260620 14:24:25.602632 18533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45613 every 8 connection(s)
I20260620 14:24:25.603612 18466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:25.603986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 18466
I20260620 14:24:25.604148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:25.604486 392 external_mini_cluster.cc:1428] Setting key 92868d790a1864fd9b3f009005505a33
I20260620 14:24:25.606889 18534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:25.612708 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb: Bootstrap starting.
Jun 20 14:24:25 dist-test-slave-87nc krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:25.615175 18534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:25.615971 18534 log.cc:824] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:25.618208 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb: No bootstrap required, opened a new log
I20260620 14:24:25.618898 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:25.605966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.0.98.62:45613)
0620 14:24:25.606435 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:25.606451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:25.606487 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:24:25.607187 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:25.607207 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:25.607557 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:25.607934 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:25.608760 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:25.609792 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:25.610677 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:25.611158 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:25.611331 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:25.613996 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:25.614003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:25.614022 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:25.614052 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:25.615804 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:25.616305 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:25.616313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:25.616319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:25.616492 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:25.616902 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:25.616905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:25.616906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:25.617289 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:25.617463 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:25.617759 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 14:24:25.618078 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":152,"threads_started":1}
I20260620 14:24:25.621923 18534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fd4370a39b4c46a149a2073f794feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } }
I20260620 14:24:25.622179 18534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:25.622260 18534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fd4370a39b4c46a149a2073f794feb, State: Initialized, Role: FOLLOWER
I20260620 14:24:25.622681 18534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fd4370a39b4c46a149a2073f794feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } }
I20260620 14:24:25.622836 18534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:25.622920 18534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:25.623041 18534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:25.624043 18534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fd4370a39b4c46a149a2073f794feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } }
I20260620 14:24:25.624382 18534 leader_election.cc:302] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76fd4370a39b4c46a149a2073f794feb; no voters:
I20260620 14:24:25.624743 18534 leader_election.cc:288] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:25.624878 18539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:25.625092 18539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [term 1 LEADER]: Becoming Leader. State: Replica: 76fd4370a39b4c46a149a2073f794feb, State: Running, Role: LEADER
I20260620 14:24:25.625430 18539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fd4370a39b4c46a149a2073f794feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } }
I20260620 14:24:25.626021 18534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:25.627246 18540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76fd4370a39b4c46a149a2073f794feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fd4370a39b4c46a149a2073f794feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } } }
I20260620 14:24:25.627365 18540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:25.627682 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76fd4370a39b4c46a149a2073f794feb. Latest consensus state: current_term: 1 leader_uuid: "76fd4370a39b4c46a149a2073f794feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fd4370a39b4c46a149a2073f794feb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } } }
I20260620 14:24:25.627955 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:25.628706 18548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:25.631412 18548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:25.637090 18548 catalog_manager.cc:1347] Generated new cluster ID: 873214655c944a3a9941b2f672d18a5f
I20260620 14:24:25.637183 18548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:25.650931 18548 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:25.652212 18548 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:25.669970 18548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb: Generated new TSK 0
I20260620 14:24:25.670708 18548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:25.706635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45613
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:25.817482 18560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:25.817764 18560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:25.817809 18560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:25.822063 18560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:25.822127 18560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:25.822188 18560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:25.827720 18560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:25.828891 18560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:25.829954 18560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:25.839270 18568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:25.839323 18565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:25.839330 18566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:25.839439 18560 server_base.cc:1034] running on GCE node
I20260620 14:24:25.840027 18560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:25.840695 18560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:25.841905 18560 hybrid_clock.cc:648] HybridClock initialized: now 1781965465841858 us; error 39 us; skew 500 ppm
Jun 20 14:24:25 dist-test-slave-87nc krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:25.844740 18560 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:25.845557 18560 webserver.cc:460] Webserver started at http://127.0.98.1:44331/ using document root <none> and password file <none>
I20260620 14:24:25.846071 18560 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:25.846129 18560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:25.846282 18560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:25.848506 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ef57432a28c42a181c242870cc6233e"
format_stamp: "Formatted at 2026-06-20 14:24:25 on dist-test-slave-87nc"
server_key: "4d66c961e1d9a3ce6937d5969b218e83"
server_key_iv: "2cd6a0d50bc0ced68e0f3040abb35063"
server_key_version: "encryptionkey@0"
I20260620 14:24:25.849012 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ef57432a28c42a181c242870cc6233e"
format_stamp: "Formatted at 2026-06-20 14:24:25 on dist-test-slave-87nc"
server_key: "4d66c961e1d9a3ce6937d5969b218e83"
server_key_iv: "2cd6a0d50bc0ced68e0f3040abb35063"
server_key_version: "encryptionkey@0"
I20260620 14:24:25.852679 18560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:24:25.855010 18575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:25.856372 18560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:25.856520 18560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "5ef57432a28c42a181c242870cc6233e"
format_stamp: "Formatted at 2026-06-20 14:24:25 on dist-test-slave-87nc"
server_key: "4d66c961e1d9a3ce6937d5969b218e83"
server_key_iv: "2cd6a0d50bc0ced68e0f3040abb35063"
server_key_version: "encryptionkey@0"
I20260620 14:24:25.856632 18560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:25.875640 18560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:25.878765 18560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:25.878999 18560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:25.879576 18560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:25.880441 18560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:25.880489 18560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:25.880556 18560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:25.880586 18560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:25.889596 18560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41719
I20260620 14:24:25.889616 18688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41719 every 8 connection(s)
I20260620 14:24:25.890518 18560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:25.892848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 18560
I20260620 14:24:25.893007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:25.893375 392 external_mini_cluster.cc:1428] Setting key 674ce34bcbf389e4431dffbcb10ba4a9
Jun 20 14:24:25 dist-test-slave-87nc krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:24:25.906344 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:25.892413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47003 (local address 127.0.98.62:45613)
0620 14:24:25.892558 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:25.892562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:25.893489 (+ 927us) server_negotiation.cc:407] Connection header received
0620 14:24:25.895233 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:25.895236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:25.895278 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:25.895348 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:25.896965 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:25.897511 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:25.899357 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:25.899598 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:25.899683 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:25.902659 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:25.902662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:25.902664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:25.902688 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:25.904270 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:25.904796 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:25.904801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:25.904803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:25.904888 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:25.905256 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:25.905262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:25.905266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:25.905464 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:25.905577 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:25.906058 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 14:24:25.906183 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:24:25.907140 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:25.892778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45613 (local address 127.0.98.1:47003)
0620 14:24:25.893319 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:25.893378 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:25.894982 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:25.895534 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:25.895546 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:25.896121 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:25.896810 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:25.896828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:25.897652 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:25.897658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:25.899222 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:25.899237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:25.899824 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:25.899832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:25.900078 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:25.900728 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:25.900754 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:25.902476 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:25.904413 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:25.904418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:25.904433 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:25.904686 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:25.905008 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:25.905013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:25.905016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:25.905155 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:25.905573 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:25.905580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:25.905827 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 14:24:25.906020 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 14:24:25.906340 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":159,"threads_started":1}
I20260620 14:24:25.908561 18689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45613
I20260620 14:24:25.908813 18689 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:25.909461 18689 heartbeater.cc:511] Master 127.0.98.62:45613 requested a full tablet report, sending...
I20260620 14:24:25.911111 18498 ts_manager.cc:194] Registered new tserver with Master: 5ef57432a28c42a181c242870cc6233e (127.0.98.1:41719)
I20260620 14:24:25.912286 18498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:25.924227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45613
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:26.035166 18694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:26.035501 18694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:26.035574 18694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:26.039901 18694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:26.039961 18694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:26.040021 18694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:26.045166 18694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:26.046134 18694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:26.047097 18694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:26.056931 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:26.056931 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:26.056945 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:26.057382 18694 server_base.cc:1034] running on GCE node
I20260620 14:24:26.057720 18694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:26.058348 18694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:26.059522 18694 hybrid_clock.cc:648] HybridClock initialized: now 1781965466059507 us; error 32 us; skew 500 ppm
Jun 20 14:24:26 dist-test-slave-87nc krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:26.062552 18694 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:26.063446 18694 webserver.cc:460] Webserver started at http://127.0.98.2:35299/ using document root <none> and password file <none>
I20260620 14:24:26.064101 18694 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:26.064174 18694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:26.064363 18694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:26.066452 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "4065d76bbaa84099a9d9f43b70df0fdb"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "edd76989834be2ea6eed354792a90f89"
server_key_iv: "8986cb1ad704006df09f311ef700e826"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.067022 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4065d76bbaa84099a9d9f43b70df0fdb"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "edd76989834be2ea6eed354792a90f89"
server_key_iv: "8986cb1ad704006df09f311ef700e826"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.071027 18694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:24:26.073550 18709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:26.074815 18694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:26.074920 18694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "4065d76bbaa84099a9d9f43b70df0fdb"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "edd76989834be2ea6eed354792a90f89"
server_key_iv: "8986cb1ad704006df09f311ef700e826"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.075021 18694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:26.104176 18694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:26.108023 18694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:26.108201 18694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:26.108775 18694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:26.109654 18694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:26.109707 18694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:26.109752 18694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:26.109779 18694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:26.119585 18694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43743
I20260620 14:24:26.119611 18822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43743 every 8 connection(s)
I20260620 14:24:26.120638 18694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:24:26 dist-test-slave-87nc krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:26.130961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 18694
I20260620 14:24:26.131095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:26.131378 392 external_mini_cluster.cc:1428] Setting key c7fd43a3a961c8c044c71f6db88325a3
I20260620 14:24:26.135901 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.122572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59347 (local address 127.0.98.62:45613)
0620 14:24:26.122715 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.122718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.123549 (+ 831us) server_negotiation.cc:407] Connection header received
0620 14:24:26.124968 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.124971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.125033 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.125195 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:26.126491 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.127172 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.128499 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.128725 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.128845 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.131418 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:26.131423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:26.131425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:26.131480 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:26.133308 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:26.133919 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:26.133926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:26.133931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:26.134030 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:26.134433 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:26.134439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:26.134444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:26.134706 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:26.134821 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.135590 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.135693 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:24:26.137212 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.122944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45613 (local address 127.0.98.2:59347)
0620 14:24:26.123397 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:26.123438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:26.124715 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:26.125374 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:26.125406 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:26.125840 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:26.126324 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.126338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.127313 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.127317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.128359 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.128369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.128890 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.128894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.129069 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.129680 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:26.129697 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:26.131245 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:26.133430 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:26.133435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:26.133449 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:26.133741 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:26.134156 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:26.134159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:26.134161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:26.134324 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:26.134893 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:26.134904 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:26.135288 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 14:24:26.135551 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 14:24:26.136005 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":132,"threads_started":1}
I20260620 14:24:26.138808 18823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45613
I20260620 14:24:26.139101 18823 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:26.139803 18823 heartbeater.cc:511] Master 127.0.98.62:45613 requested a full tablet report, sending...
I20260620 14:24:26.140988 18498 ts_manager.cc:194] Registered new tserver with Master: 4065d76bbaa84099a9d9f43b70df0fdb (127.0.98.2:43743)
I20260620 14:24:26.141577 18498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59347
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:26.162683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45613
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:26.274654 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:26.274933 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:26.274977 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:26.279310 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:26.279376 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:26.279436 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:26.284986 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:26.286011 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:26.287007 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:26.296592 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:26.296609 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:26.296682 18828 server_base.cc:1034] running on GCE node
W20260620 14:24:26.296605 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:26.297221 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:26.297888 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:26.299127 18828 hybrid_clock.cc:648] HybridClock initialized: now 1781965466299095 us; error 45 us; skew 500 ppm
Jun 20 14:24:26 dist-test-slave-87nc krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:26.301883 18828 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:26.302806 18828 webserver.cc:460] Webserver started at http://127.0.98.3:44515/ using document root <none> and password file <none>
I20260620 14:24:26.303377 18828 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:26.303493 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:26.303725 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:26.305809 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "5bdee2f03f8f4b0894e2104b1a86fced"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "68dad9fda41bf2445b34bd466cd122ad"
server_key_iv: "d92a31232d172786c4c49a88712d346c"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.306372 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5bdee2f03f8f4b0894e2104b1a86fced"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "68dad9fda41bf2445b34bd466cd122ad"
server_key_iv: "d92a31232d172786c4c49a88712d346c"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.310194 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:24:26.312702 18843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:26.313911 18828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:26.314050 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "5bdee2f03f8f4b0894e2104b1a86fced"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "68dad9fda41bf2445b34bd466cd122ad"
server_key_iv: "d92a31232d172786c4c49a88712d346c"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.314167 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:26.334182 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:26.337247 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:26.337481 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:26.338104 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:26.338950 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:26.339030 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:26.339097 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:26.339151 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:26.348557 18828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34269
I20260620 14:24:26.348583 18956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34269 every 8 connection(s)
I20260620 14:24:26.349427 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:24:26 dist-test-slave-87nc krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:26.359367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 18828
I20260620 14:24:26.359525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:26.359900 392 external_mini_cluster.cc:1428] Setting key 42f0f3d78e31d86e711e976c46fb0887
I20260620 14:24:26.363371 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.351249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49221 (local address 127.0.98.62:45613)
0620 14:24:26.351383 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.351391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.352187 (+ 796us) server_negotiation.cc:407] Connection header received
0620 14:24:26.353485 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.353488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.353548 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.353631 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:26.354903 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.355437 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.356658 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.356884 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.357001 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.359579 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:26.359584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:26.359589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:26.359620 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:26.361263 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:26.361858 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:26.361860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:26.361861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:26.361953 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:26.362321 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:26.362324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:26.362326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:26.362515 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:26.362625 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.363104 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.363218 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:24:26.364221 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.351573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45613 (local address 127.0.98.3:49221)
0620 14:24:26.352036 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:26.352080 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:26.353266 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:26.353819 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:26.353826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:26.354186 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:26.354759 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.354775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.355571 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.355574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.356531 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.356539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.357039 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.357044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.357205 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.357814 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:26.357830 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:26.359312 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:26.361387 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:26.361391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:26.361405 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:26.361767 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:26.362051 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:26.362054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:26.362056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:26.362210 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:26.362621 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:26.362627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:26.362864 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 14:24:26.363065 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 14:24:26.363374 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":139,"threads_started":1}
I20260620 14:24:26.365335 18957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45613
I20260620 14:24:26.365571 18957 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:26.366058 18957 heartbeater.cc:511] Master 127.0.98.62:45613 requested a full tablet report, sending...
I20260620 14:24:26.367061 18498 ts_manager.cc:194] Registered new tserver with Master: 5bdee2f03f8f4b0894e2104b1a86fced (127.0.98.3:34269)
I20260620 14:24:26.367633 18498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49221
I20260620 14:24:26.374086 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:26.383730 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.376113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.0.98.62:45613)
0620 14:24:26.376259 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.376264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.376433 (+ 169us) server_negotiation.cc:407] Connection header received
0620 14:24:26.376518 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.376521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.376561 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.376626 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:26.377294 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.377811 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.378759 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.378971 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.379050 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.380704 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:26.380708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:26.380710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:26.380732 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:26.382155 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:26.382612 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:26.382616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:26.382618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:26.382704 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:26.382996 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:26.382999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:26.383000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:26.383194 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:26.383304 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.383537 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.383612 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:24:26.386622 18498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:26.389683 18498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:26.404474 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.398520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.0.98.1:41719)
0620 14:24:26.399798 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.399817 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.399836 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:24:26.399954 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.399960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.400151 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.400590 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:26.400750 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.401476 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.402983 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.403840 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.404038 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.404198 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.404267 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.404342 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":70,"threads_started":1}
I20260620 14:24:26.404518 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.398376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41719 (local address 127.0.0.1:53950)
0620 14:24:26.398750 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:26.398801 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:26.398938 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:26.400311 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:26.400314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:26.400331 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:26.400447 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.400455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.401607 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.401614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.402852 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.402861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.403981 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.403985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.404095 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.404136 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:24:26.404249 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:24:26.404367 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":113,"thread_start_us":97,"threads_started":1}
I20260620 14:24:26.405135 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.399119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43743 (local address 127.0.0.1:56706)
0620 14:24:26.399785 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:26.399798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:26.399881 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:26.400464 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:26.400467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:26.400485 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:26.400566 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.400571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.401983 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.401996 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.402970 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.402980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.403859 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.403863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.403974 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.404838 (+ 864us) client_negotiation.cc:769] Sending connection context
0620 14:24:26.404910 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:24:26.404997 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260620 14:24:26.405616 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.398752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34269 (local address 127.0.0.1:54052)
0620 14:24:26.399483 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:26.399508 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:26.399597 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:26.400215 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:26.400221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:26.400243 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:26.400436 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.400443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.401983 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.401986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.404013 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.404021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.405241 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.405245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.405352 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.405375 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:24:26.405437 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:24:26.405505 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":868,"thread_start_us":92,"threads_started":1}
I20260620 14:24:26.405612 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.399297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.0.98.2:43743)
0620 14:24:26.399717 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.399732 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.399887 (+ 155us) server_negotiation.cc:407] Connection header received
0620 14:24:26.399981 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.399985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.400151 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.400301 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:26.400696 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.401829 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.403114 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.403690 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.403860 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.404023 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.405347 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.405462 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":114,"threads_started":1}
I20260620 14:24:26.406107 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.399199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.0.98.3:34269)
0620 14:24:26.399642 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.399655 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.399679 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:24:26.399757 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.399762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.400051 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.400225 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:26.400565 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.401828 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.404106 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.404957 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.405049 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.405173 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.405873 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.405967 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":129,"threads_started":1}
I20260620 14:24:26.407601 18623 tablet_service.cc:1467] Processing CreateTablet for tablet d45c5186e5a84c8fa871e430a7373498 (DEFAULT_TABLE table=test-table [id=f027604ec40c4f61bdfdfdb4aba08184]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:26.408627 18623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45c5186e5a84c8fa871e430a7373498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:26.408339 18757 tablet_service.cc:1467] Processing CreateTablet for tablet d45c5186e5a84c8fa871e430a7373498 (DEFAULT_TABLE table=test-table [id=f027604ec40c4f61bdfdfdb4aba08184]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:26.408682 18891 tablet_service.cc:1467] Processing CreateTablet for tablet d45c5186e5a84c8fa871e430a7373498 (DEFAULT_TABLE table=test-table [id=f027604ec40c4f61bdfdfdb4aba08184]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:26.409379 18757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45c5186e5a84c8fa871e430a7373498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:26.409605 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45c5186e5a84c8fa871e430a7373498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:26.414800 18974 tablet_bootstrap.cc:492] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e: Bootstrap starting.
I20260620 14:24:26.416782 18975 tablet_bootstrap.cc:492] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: Bootstrap starting.
I20260620 14:24:26.417090 18974 tablet_bootstrap.cc:654] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:26.417850 18974 log.cc:824] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:26.418804 18975 tablet_bootstrap.cc:654] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:26.419775 18974 tablet_bootstrap.cc:492] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e: No bootstrap required, opened a new log
I20260620 14:24:26.420002 18974 ts_tablet_manager.cc:1397] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:24:26.420104 18976 tablet_bootstrap.cc:492] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: Bootstrap starting.
I20260620 14:24:26.420109 18975 log.cc:824] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:26.422123 18975 tablet_bootstrap.cc:492] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: No bootstrap required, opened a new log
I20260620 14:24:26.422201 18976 tablet_bootstrap.cc:654] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:26.422323 18975 ts_tablet_manager.cc:1397] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:24:26.423384 18976 log.cc:824] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:26.423672 18974 raft_consensus.cc:348] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } }
I20260620 14:24:26.423992 18974 raft_consensus.cc:374] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:26.424053 18974 raft_consensus.cc:729] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef57432a28c42a181c242870cc6233e, State: Initialized, Role: FOLLOWER
I20260620 14:24:26.424508 18974 consensus_queue.cc:260] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } }
I20260620 14:24:26.425137 18689 heartbeater.cc:503] Master 127.0.98.62:45613 was elected leader, sending a full tablet report...
I20260620 14:24:26.425405 18976 tablet_bootstrap.cc:492] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: No bootstrap required, opened a new log
I20260620 14:24:26.425539 18974 ts_tablet_manager.cc:1428] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:24:26.425577 18976 ts_tablet_manager.cc:1397] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:24:26.426265 18975 raft_consensus.cc:348] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } }
I20260620 14:24:26.426637 18975 raft_consensus.cc:374] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:26.426689 18975 raft_consensus.cc:729] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4065d76bbaa84099a9d9f43b70df0fdb, State: Initialized, Role: FOLLOWER
I20260620 14:24:26.427141 18975 consensus_queue.cc:260] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } }
I20260620 14:24:26.427974 18823 heartbeater.cc:503] Master 127.0.98.62:45613 was elected leader, sending a full tablet report...
I20260620 14:24:26.428186 18975 ts_tablet_manager.cc:1428] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:24:26.429301 18976 raft_consensus.cc:348] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } }
I20260620 14:24:26.429631 18976 raft_consensus.cc:374] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:26.429694 18976 raft_consensus.cc:729] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdee2f03f8f4b0894e2104b1a86fced, State: Initialized, Role: FOLLOWER
I20260620 14:24:26.430167 18976 consensus_queue.cc:260] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } }
I20260620 14:24:26.430809 18957 heartbeater.cc:503] Master 127.0.98.62:45613 was elected leader, sending a full tablet report...
I20260620 14:24:26.431162 18976 ts_tablet_manager.cc:1428] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:24:26.496040 18980 raft_consensus.cc:482] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:26.496275 18980 raft_consensus.cc:504] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } }
I20260620 14:24:26.497419 18980 leader_election.cc:288] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4065d76bbaa84099a9d9f43b70df0fdb (127.0.98.2:43743), 5bdee2f03f8f4b0894e2104b1a86fced (127.0.98.3:34269)
I20260620 14:24:26.501752 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.497886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58977 (local address 127.0.98.2:43743)
0620 14:24:26.498026 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.498030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.498300 (+ 270us) server_negotiation.cc:407] Connection header received
0620 14:24:26.498437 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.498441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.498490 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.498567 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:26.498893 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.499369 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.500718 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.501114 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.501193 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.501276 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.501561 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.501626 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:24:26.501789 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.497897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55359 (local address 127.0.98.3:34269)
0620 14:24:26.498046 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.498052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.498178 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:24:26.498290 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.498293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.498357 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.498443 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:26.498751 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.499389 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.500702 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.501148 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.501295 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.501361 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.501571 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.501643 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:24:26.501806 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.497682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34269 (local address 127.0.98.1:55359)
0620 14:24:26.498064 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:26.498078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:26.498186 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:26.498457 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:26.498460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:26.498475 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:26.498615 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.498622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.499539 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.499541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.500554 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.500563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.501311 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.501315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.501410 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.501448 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:24:26.501541 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:24:26.501616 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":105,"thread_start_us":129,"threads_started":1}
I20260620 14:24:26.501817 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.497738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43743 (local address 127.0.98.1:58977)
0620 14:24:26.498170 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:26.498186 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:26.498308 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:26.498606 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:26.498608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:26.498623 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:26.498774 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.498782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.499497 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.499507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.500556 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:26.500566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.501248 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.501255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:26.501367 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.501445 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 14:24:26.501545 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:24:26.501629 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":61,"thread_start_us":121,"threads_started":1}
I20260620 14:24:26.502463 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45c5186e5a84c8fa871e430a7373498" candidate_uuid: "5ef57432a28c42a181c242870cc6233e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" is_pre_election: true
I20260620 14:24:26.502535 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45c5186e5a84c8fa871e430a7373498" candidate_uuid: "5ef57432a28c42a181c242870cc6233e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" is_pre_election: true
I20260620 14:24:26.502808 18777 raft_consensus.cc:2454] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef57432a28c42a181c242870cc6233e in term 0.
I20260620 14:24:26.502797 18911 raft_consensus.cc:2454] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef57432a28c42a181c242870cc6233e in term 0.
I20260620 14:24:26.503302 18578 leader_election.cc:302] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4065d76bbaa84099a9d9f43b70df0fdb, 5ef57432a28c42a181c242870cc6233e; no voters:
I20260620 14:24:26.503585 18980 raft_consensus.cc:2781] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:26.503666 18980 raft_consensus.cc:482] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:26.503724 18980 raft_consensus.cc:3037] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:26.504864 18980 raft_consensus.cc:504] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } }
I20260620 14:24:26.505239 18980 leader_election.cc:288] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [CANDIDATE]: Term 1 election: Requested vote from peers 4065d76bbaa84099a9d9f43b70df0fdb (127.0.98.2:43743), 5bdee2f03f8f4b0894e2104b1a86fced (127.0.98.3:34269)
I20260620 14:24:26.505615 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45c5186e5a84c8fa871e430a7373498" candidate_uuid: "5ef57432a28c42a181c242870cc6233e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4065d76bbaa84099a9d9f43b70df0fdb"
I20260620 14:24:26.505605 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45c5186e5a84c8fa871e430a7373498" candidate_uuid: "5ef57432a28c42a181c242870cc6233e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bdee2f03f8f4b0894e2104b1a86fced"
I20260620 14:24:26.505779 18911 raft_consensus.cc:3037] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:26.505766 18777 raft_consensus.cc:3037] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:26.507179 18777 raft_consensus.cc:2454] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef57432a28c42a181c242870cc6233e in term 1.
I20260620 14:24:26.507179 18911 raft_consensus.cc:2454] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef57432a28c42a181c242870cc6233e in term 1.
I20260620 14:24:26.507534 18578 leader_election.cc:302] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4065d76bbaa84099a9d9f43b70df0fdb, 5ef57432a28c42a181c242870cc6233e; no voters:
I20260620 14:24:26.507731 18980 raft_consensus.cc:2781] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:26.508035 18980 raft_consensus.cc:686] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 1 LEADER]: Becoming Leader. State: Replica: 5ef57432a28c42a181c242870cc6233e, State: Running, Role: LEADER
I20260620 14:24:26.508394 18980 consensus_queue.cc:237] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } }
I20260620 14:24:26.511529 18498 catalog_manager.cc:5495] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ef57432a28c42a181c242870cc6233e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5ef57432a28c42a181c242870cc6233e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef57432a28c42a181c242870cc6233e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:26.526782 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.523265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.0.98.1:41719)
0620 14:24:26.523388 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.523392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.523412 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:24:26.523532 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.523534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.523589 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.523670 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:26.523963 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.524452 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.525254 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.525462 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.525535 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.525922 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:26.526317 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:26.526440 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.526561 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.526595 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:24:26.532661 18777 raft_consensus.cc:1261] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [term 1 FOLLOWER]: Refusing update from remote peer 5ef57432a28c42a181c242870cc6233e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:26.532670 18911 raft_consensus.cc:1261] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [term 1 FOLLOWER]: Refusing update from remote peer 5ef57432a28c42a181c242870cc6233e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:26.533457 18985 consensus_queue.cc:1035] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bdee2f03f8f4b0894e2104b1a86fced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:26.533699 18980 consensus_queue.cc:1035] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4065d76bbaa84099a9d9f43b70df0fdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:26.539924 18988 mvcc.cc:205] Tried to move back new op lower bound from 7298930550910685184 to 7298930550821146624. Current Snapshot: MvccSnapshot[applied={T|T < 7298930550910685184}]
I20260620 14:24:26.540300 18989 mvcc.cc:205] Tried to move back new op lower bound from 7298930550910685184 to 7298930550821146624. Current Snapshot: MvccSnapshot[applied={T|T < 7298930550910685184}]
I20260620 14:24:26.551155 18497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:26.551364 18497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:26.553763 18497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76fd4370a39b4c46a149a2073f794feb: Sending DeleteTablet for 3 replicas of tablet d45c5186e5a84c8fa871e430a7373498
I20260620 14:24:26.554575 18623 tablet_service.cc:1514] Processing DeleteTablet for tablet d45c5186e5a84c8fa871e430a7373498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53950
I20260620 14:24:26.554577 18757 tablet_service.cc:1514] Processing DeleteTablet for tablet d45c5186e5a84c8fa871e430a7373498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56706
I20260620 14:24:26.554593 18891 tablet_service.cc:1514] Processing DeleteTablet for tablet d45c5186e5a84c8fa871e430a7373498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54052
I20260620 14:24:26.555099 19000 tablet_replica.cc:330] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: stopping tablet replica
I20260620 14:24:26.555099 18998 tablet_replica.cc:330] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: stopping tablet replica
I20260620 14:24:26.555424 18998 raft_consensus.cc:2229] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:26.555517 18999 tablet_replica.cc:330] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e: stopping tablet replica
I20260620 14:24:26.555737 18998 raft_consensus.cc:2258] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:26.555791 18999 raft_consensus.cc:2229] T d45c5186e5a84c8fa871e430a7373498 P 5ef57432a28c42a181c242870cc6233e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:26.556067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 18560
I20260620 14:24:26.557858 18998 ts_tablet_manager.cc:1905] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:26.558218 19000 raft_consensus.cc:2229] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:26.558525 19000 raft_consensus.cc:2258] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:26.559815 19000 ts_tablet_manager.cc:1905] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:26.561266 18998 ts_tablet_manager.cc:1918] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:26.561378 18998 log.cc:1196] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/d45c5186e5a84c8fa871e430a7373498
I20260620 14:24:26.561707 18998 ts_tablet_manager.cc:1939] T d45c5186e5a84c8fa871e430a7373498 P 5bdee2f03f8f4b0894e2104b1a86fced: Deleting consensus metadata
I20260620 14:24:26.561986 19000 ts_tablet_manager.cc:1918] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:26.562093 19000 log.cc:1196] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/d45c5186e5a84c8fa871e430a7373498
I20260620 14:24:26.562377 19000 ts_tablet_manager.cc:1939] T d45c5186e5a84c8fa871e430a7373498 P 4065d76bbaa84099a9d9f43b70df0fdb: Deleting consensus metadata
I20260620 14:24:26.562973 18485 catalog_manager.cc:4841] TS 5bdee2f03f8f4b0894e2104b1a86fced (127.0.98.3:34269): tablet d45c5186e5a84c8fa871e430a7373498 (table test-table [id=f027604ec40c4f61bdfdfdb4aba08184]) successfully deleted
I20260620 14:24:26.563427 18484 catalog_manager.cc:4841] TS 4065d76bbaa84099a9d9f43b70df0fdb (127.0.98.2:43743): tablet d45c5186e5a84c8fa871e430a7373498 (table test-table [id=f027604ec40c4f61bdfdfdb4aba08184]) successfully deleted
W20260620 14:24:26.563846 18484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41719 (error 108)
I20260620 14:24:26.564131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 18694
I20260620 14:24:26.565414 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.565063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41719 (local address 127.0.0.1:49078)
0620 14:24:26.565225 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:26.565311 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:24:26.565582 18484 catalog_manager.cc:4584] TS 5ef57432a28c42a181c242870cc6233e (127.0.98.1:41719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d45c5186e5a84c8fa871e430a7373498: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41719: connect: Connection refused (error 111)
I20260620 14:24:26.569614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 18828
I20260620 14:24:26.575126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 18466
2026-06-20T14:24:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2321 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:26Z Disabled control of system clock
I20260620 14:24:26.611747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45561
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:24:26.728291 19005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:26.728556 19005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:26.728592 19005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:26.732684 19005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:26.732734 19005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:26.732749 19005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:26.732801 19005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:26.732832 19005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:26.737484 19005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:26.738554 19005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:26.739960 19005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:26.748453 19005 server_base.cc:1034] running on GCE node
W20260620 14:24:26.748382 19010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:26.748405 19013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:26.748363 19011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:26.749266 19005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:26.750489 19005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:26.751652 19005 hybrid_clock.cc:648] HybridClock initialized: now 1781965466751630 us; error 38 us; skew 500 ppm
I20260620 14:24:26.753567 19005 webserver.cc:460] Webserver started at http://127.0.98.62:32775/ using document root <none> and password file <none>
I20260620 14:24:26.754158 19005 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:26.754220 19005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:26.754459 19005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:26.756623 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "47ef48c1ab8e442ea8bee0c58df16c92"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "35a626e15d997dee2e307a3b626799fa"
server_key_iv: "449814ec64fdd8418a815f865797a6b6"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.757203 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "47ef48c1ab8e442ea8bee0c58df16c92"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "35a626e15d997dee2e307a3b626799fa"
server_key_iv: "449814ec64fdd8418a815f865797a6b6"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.760856 19005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:26.763116 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:26.764382 19005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:26.764539 19005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "47ef48c1ab8e442ea8bee0c58df16c92"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "35a626e15d997dee2e307a3b626799fa"
server_key_iv: "449814ec64fdd8418a815f865797a6b6"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.764655 19005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:26.773851 19005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:26.774459 19005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:26.774657 19005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:26.781559 19005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45561
I20260620 14:24:26.781565 19071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45561 every 8 connection(s)
I20260620 14:24:26.782461 19005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:26.785646 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:26.788625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 19005
I20260620 14:24:26.788779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:26.789052 392 external_mini_cluster.cc:1428] Setting key 1f8c0ccb77b357c4041a5011484db3d0
I20260620 14:24:26.791545 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92: Bootstrap starting.
I20260620 14:24:26.793738 19072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:26.794479 19072 log.cc:824] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:26.796520 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92: No bootstrap required, opened a new log
I20260620 14:24:26.798384 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.790516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.0.98.62:45561)
0620 14:24:26.791150 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:26.791165 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:26.791200 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:24:26.791960 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:26.791980 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:26.792372 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:26.792708 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:26.793145 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.794145 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.795095 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:26.795562 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:26.795743 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:26.796654 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:26.796660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:26.796677 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:26.797085 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:26.797144 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:26.797170 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:26.797295 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:26.797442 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:24:26.797696 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":130,"threads_started":1}
I20260620 14:24:26.800549 19072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef48c1ab8e442ea8bee0c58df16c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45561 } }
I20260620 14:24:26.800843 19072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:26.800906 19072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ef48c1ab8e442ea8bee0c58df16c92, State: Initialized, Role: FOLLOWER
I20260620 14:24:26.801447 19072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef48c1ab8e442ea8bee0c58df16c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45561 } }
I20260620 14:24:26.801638 19072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:26.801708 19072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:26.801818 19072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:26.803134 19072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef48c1ab8e442ea8bee0c58df16c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45561 } }
I20260620 14:24:26.803520 19072 leader_election.cc:302] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47ef48c1ab8e442ea8bee0c58df16c92; no voters:
I20260620 14:24:26.803927 19072 leader_election.cc:288] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:26.804078 19077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:26.804327 19077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [term 1 LEADER]: Becoming Leader. State: Replica: 47ef48c1ab8e442ea8bee0c58df16c92, State: Running, Role: LEADER
I20260620 14:24:26.804657 19077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef48c1ab8e442ea8bee0c58df16c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45561 } }
I20260620 14:24:26.805208 19072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:26.806636 19078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47ef48c1ab8e442ea8bee0c58df16c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef48c1ab8e442ea8bee0c58df16c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45561 } } }
I20260620 14:24:26.806762 19078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:26.807053 19079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47ef48c1ab8e442ea8bee0c58df16c92. Latest consensus state: current_term: 1 leader_uuid: "47ef48c1ab8e442ea8bee0c58df16c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef48c1ab8e442ea8bee0c58df16c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45561 } } }
I20260620 14:24:26.807155 19079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92 [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:26.810096 19092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:26.810192 19092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:26.810282 19089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:26.811175 19089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:26.816668 19089 catalog_manager.cc:1347] Generated new cluster ID: 94abb0c00be9491db4b1744e91bc1808
I20260620 14:24:26.816735 19089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:26.822649 19089 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:26.823549 19089 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:26.828279 19089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92: Generated new TSK 0
I20260620 14:24:26.829106 19089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:26.837600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45561
--builtin_ntp_servers=127.0.98.20:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:24:26.958636 19096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:26.958969 19096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:26.959023 19096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:26.963292 19096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:26.963395 19096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:26.963487 19096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:26.968650 19096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:26.969779 19096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:26.971164 19096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:26.981230 19101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:26.981200 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:26.981262 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:26.981564 19096 server_base.cc:1034] running on GCE node
I20260620 14:24:26.981997 19096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:26.982645 19096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:26.983839 19096 hybrid_clock.cc:648] HybridClock initialized: now 1781965466983795 us; error 51 us; skew 500 ppm
I20260620 14:24:26.985726 19096 webserver.cc:460] Webserver started at http://127.0.98.1:33091/ using document root <none> and password file <none>
I20260620 14:24:26.986294 19096 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:26.986363 19096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:26.986625 19096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:26.989184 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "591437615c4f448eb72c87ab6ac0e357"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "35b03726aaba7777fc55f613a43b5711"
server_key_iv: "7abda6f66a5d5ca9f9c6fa6e8c963e75"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.989737 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "591437615c4f448eb72c87ab6ac0e357"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "35b03726aaba7777fc55f613a43b5711"
server_key_iv: "7abda6f66a5d5ca9f9c6fa6e8c963e75"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.993453 19096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:24:26.995859 19110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:26.997118 19096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:26.997215 19096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "591437615c4f448eb72c87ab6ac0e357"
format_stamp: "Formatted at 2026-06-20 14:24:26 on dist-test-slave-87nc"
server_key: "35b03726aaba7777fc55f613a43b5711"
server_key_iv: "7abda6f66a5d5ca9f9c6fa6e8c963e75"
server_key_version: "encryptionkey@0"
I20260620 14:24:26.997303 19096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:27.016161 19096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:27.016822 19096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:27.016975 19096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:27.017495 19096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:27.018378 19096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:27.018448 19096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.018505 19096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:27.018520 19096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.027586 19096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40441
I20260620 14:24:27.027609 19223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40441 every 8 connection(s)
I20260620 14:24:27.028600 19096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:27.029155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 19096
I20260620 14:24:27.029280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:27.029606 392 external_mini_cluster.cc:1428] Setting key 1f9a1d0c80905d5dd67fdc398e117d3b
I20260620 14:24:27.031759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45561
--builtin_ntp_servers=127.0.98.20:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:24:27.039017 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.031937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37457 (local address 127.0.98.62:45561)
0620 14:24:27.033214 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.033222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.033240 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:24:27.033744 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.033749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.033808 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.033906 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:27.034795 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.035686 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.036819 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.037221 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.037350 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.038459 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:27.038465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:27.038470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:27.038486 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:27.038509 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:27.038521 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:27.038609 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.038751 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.038819 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180}
I20260620 14:24:27.039835 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.031391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45561 (local address 127.0.98.1:37457)
0620 14:24:27.032250 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.032299 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:27.033545 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:27.034060 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:27.034068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:27.034151 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:27.034635 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.034647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.035839 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.035843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.036691 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.036700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.037377 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.037379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.037520 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.038185 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:27.038187 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:27.038226 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:27.038285 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:27.038613 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:27.038619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:27.038623 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:24:27.038711 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:24:27.038921 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":136,"threads_started":1}
I20260620 14:24:27.041093 19224 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45561
I20260620 14:24:27.041337 19224 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:27.041879 19224 heartbeater.cc:511] Master 127.0.98.62:45561 requested a full tablet report, sending...
I20260620 14:24:27.043356 19036 ts_manager.cc:194] Registered new tserver with Master: 591437615c4f448eb72c87ab6ac0e357 (127.0.98.1:40441)
I20260620 14:24:27.044581 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37457
W20260620 14:24:27.145078 19227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:27.145367 19227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:27.145447 19227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:27.149616 19227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:27.149701 19227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:27.149793 19227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:27.154528 19227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:27.155633 19227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:27.157068 19227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:27.166752 19232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:27.166846 19233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:27.166772 19235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:27.167030 19227 server_base.cc:1034] running on GCE node
I20260620 14:24:27.167723 19227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:27.168414 19227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:27.169576 19227 hybrid_clock.cc:648] HybridClock initialized: now 1781965467169562 us; error 28 us; skew 500 ppm
I20260620 14:24:27.171384 19227 webserver.cc:460] Webserver started at http://127.0.98.2:35711/ using document root <none> and password file <none>
I20260620 14:24:27.172067 19227 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:27.172134 19227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:27.172377 19227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:27.174458 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "c1a6d2257ba04c1bbcd4889159636d92"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-87nc"
server_key: "838727c4b48bceb89be4e7793f4bd64f"
server_key_iv: "3b6faf8b2107e24a79f915202b20fefd"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.175029 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c1a6d2257ba04c1bbcd4889159636d92"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-87nc"
server_key: "838727c4b48bceb89be4e7793f4bd64f"
server_key_iv: "3b6faf8b2107e24a79f915202b20fefd"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.178781 19227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:24:27.181283 19241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.182463 19227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:24:27.182590 19227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "c1a6d2257ba04c1bbcd4889159636d92"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-87nc"
server_key: "838727c4b48bceb89be4e7793f4bd64f"
server_key_iv: "3b6faf8b2107e24a79f915202b20fefd"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.182704 19227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:27.193984 19227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:27.194638 19227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:27.194832 19227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:27.195405 19227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:27.196429 19227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:27.196475 19227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.196543 19227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:27.196574 19227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.205834 19227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40433
I20260620 14:24:27.205857 19354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40433 every 8 connection(s)
I20260620 14:24:27.206852 19227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:27.208343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 19227
I20260620 14:24:27.208464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:27.208798 392 external_mini_cluster.cc:1428] Setting key a9ad0dee9ea1e492b1cecd531561fc65
I20260620 14:24:27.210912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45561
--builtin_ntp_servers=127.0.98.20:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:24:27.221030 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.208750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40243 (local address 127.0.98.62:45561)
0620 14:24:27.208921 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.208926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.210133 (+ 1207us) server_negotiation.cc:407] Connection header received
0620 14:24:27.211685 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.211688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.211742 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.211850 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:27.213126 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.213646 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.215524 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.215736 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.215920 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.220455 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:27.220458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:27.220460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:27.220473 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:27.220494 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:27.220502 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:27.220561 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.220774 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.220830 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:24:27.222395 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.209261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45561 (local address 127.0.98.2:40243)
0620 14:24:27.209979 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.210021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:27.211465 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:27.212066 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:27.212078 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:27.212212 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:27.212955 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.212977 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.213778 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.213784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.215396 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.215412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.215962 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.215969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.216220 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.220097 (+ 3877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:27.220101 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:27.220168 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:27.220273 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:27.220596 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:27.220608 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:27.220616 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:24:27.220775 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 14:24:27.221260 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":114,"threads_started":1}
I20260620 14:24:27.223664 19355 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45561
I20260620 14:24:27.223963 19355 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:27.224502 19355 heartbeater.cc:511] Master 127.0.98.62:45561 requested a full tablet report, sending...
I20260620 14:24:27.225616 19036 ts_manager.cc:194] Registered new tserver with Master: c1a6d2257ba04c1bbcd4889159636d92 (127.0.98.2:40433)
I20260620 14:24:27.226356 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40243
W20260620 14:24:27.325275 19358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:27.325533 19358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:27.325572 19358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:27.329628 19358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:27.329684 19358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:27.329738 19358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:27.334445 19358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:27.335589 19358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:27.337146 19358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:27.347114 19358 server_base.cc:1034] running on GCE node
W20260620 14:24:27.347064 19364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:27.347044 19363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:27.347159 19366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:27.347661 19358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:27.348301 19358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:27.349526 19358 hybrid_clock.cc:648] HybridClock initialized: now 1781965467349502 us; error 35 us; skew 500 ppm
I20260620 14:24:27.351343 19358 webserver.cc:460] Webserver started at http://127.0.98.3:39977/ using document root <none> and password file <none>
I20260620 14:24:27.352010 19358 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:27.352110 19358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:27.352344 19358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:27.354481 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "b94e3959235d4332adff4c0d4a69140a"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-87nc"
server_key: "743873d0aff4730e3e7bebf24016ea26"
server_key_iv: "13c8a38388919ec02d0070ae054e3ec3"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.355080 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b94e3959235d4332adff4c0d4a69140a"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-87nc"
server_key: "743873d0aff4730e3e7bebf24016ea26"
server_key_iv: "13c8a38388919ec02d0070ae054e3ec3"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.358738 19358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:24:27.361344 19372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.362584 19358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:27.362737 19358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "b94e3959235d4332adff4c0d4a69140a"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-87nc"
server_key: "743873d0aff4730e3e7bebf24016ea26"
server_key_iv: "13c8a38388919ec02d0070ae054e3ec3"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.362869 19358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:27.392071 19358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:27.392813 19358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:27.393036 19358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:27.393644 19358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:27.394600 19358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:27.394680 19358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.394750 19358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:27.394806 19358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.403714 19358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34271
I20260620 14:24:27.403792 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34271 every 8 connection(s)
I20260620 14:24:27.404707 19358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:27.408169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 19358
I20260620 14:24:27.408279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:27.408547 392 external_mini_cluster.cc:1428] Setting key 5e1259fa85de59241451c1d86a3cc00c
I20260620 14:24:27.414510 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.406734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57317 (local address 127.0.98.62:45561)
0620 14:24:27.406890 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.406896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.408153 (+ 1257us) server_negotiation.cc:407] Connection header received
0620 14:24:27.409921 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.409924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.409969 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.410038 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:27.411146 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.411655 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.412798 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.413018 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.413094 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.414141 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:27.414144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:27.414146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:27.414156 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:27.414171 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:27.414177 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:27.414231 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.414363 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.414404 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:24:27.415421 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.407280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45561 (local address 127.0.98.3:57317)
0620 14:24:27.408005 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.408047 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:27.409682 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:27.410220 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:27.410229 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:27.410331 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:27.411025 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.411040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.411806 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.411809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.412698 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.412707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.413149 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.413151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.413284 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.413918 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:27.413920 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:27.413958 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:27.414019 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:27.414250 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:27.414256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:27.414259 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:27.414348 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:24:27.414601 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
I20260620 14:24:27.416575 19486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45561
I20260620 14:24:27.416807 19486 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:27.417303 19486 heartbeater.cc:511] Master 127.0.98.62:45561 requested a full tablet report, sending...
I20260620 14:24:27.418289 19036 ts_manager.cc:194] Registered new tserver with Master: b94e3959235d4332adff4c0d4a69140a (127.0.98.3:34271)
I20260620 14:24:27.418843 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57317
I20260620 14:24:27.422634 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:27.428458 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.424496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.0.98.62:45561)
0620 14:24:27.424644 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.424650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.424815 (+ 165us) server_negotiation.cc:407] Connection header received
0620 14:24:27.424967 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.424970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.425018 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.425105 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:27.425382 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.425879 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.426724 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.426946 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.427055 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.427976 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:27.427986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:27.427988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:27.427999 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:27.428016 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:27.428023 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:27.428153 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.428294 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.428331 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:24:27.431128 19036 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:27.434090 19036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:27.449024 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.443887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.0.98.2:40433)
0620 14:24:27.444416 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.444425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.444443 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:24:27.444534 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.444538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.444708 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.444874 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:27.445087 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.446145 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.447436 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.448271 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.448387 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.448573 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.448745 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.448863 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":118,"threads_started":1}
I20260620 14:24:27.449050 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.442538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34271 (local address 127.0.0.1:54428)
0620 14:24:27.443178 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.443193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:27.443321 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:27.444240 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:27.444245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:27.444268 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:27.444424 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.444432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.445835 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.445838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.447429 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.447440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.448650 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.448654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.448749 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.448773 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:24:27.448845 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:24:27.448923 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":123,"thread_start_us":85,"threads_started":1}
I20260620 14:24:27.449321 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.442460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40441 (local address 127.0.0.1:42474)
0620 14:24:27.442849 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.442918 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:27.443861 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:27.444561 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:27.444564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:27.444585 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:27.444657 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.444661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.445744 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.445751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.447450 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.447456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.448785 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.448789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.448865 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.448886 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:24:27.448954 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:24:27.449014 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":64,"thread_start_us":94,"threads_started":1}
I20260620 14:24:27.449011 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.443706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40433 (local address 127.0.0.1:46518)
0620 14:24:27.444075 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.444087 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:27.444167 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:27.444844 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:27.444847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:27.444864 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:27.444954 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.444960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.446263 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.446266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.447303 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.447312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.448415 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.448419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.448544 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.448587 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:24:27.448696 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:24:27.448820 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":52,"thread_start_us":116,"threads_started":1}
I20260620 14:24:27.449716 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.442610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.0.98.1:40441)
0620 14:24:27.443051 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.443063 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.443091 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:24:27.444071 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.444077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.444260 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.444408 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:27.444779 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.445622 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.447859 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.448442 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.448544 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.448727 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.449095 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.449551 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":102,"threads_started":1}
I20260620 14:24:27.449735 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.443296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.0.98.3:34271)
0620 14:24:27.443689 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.443700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.443721 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:24:27.443824 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.443832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.444045 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.444245 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:27.444544 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.445743 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.447556 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.448288 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.448397 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.448573 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.449362 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.449535 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":179,"threads_started":1}
I20260620 14:24:27.452487 19420 tablet_service.cc:1467] Processing CreateTablet for tablet c9dea4f0a48f4315bb9f9b81e4f252e9 (DEFAULT_TABLE table=test-table [id=11bbdf22f2584acbb96ef93d172be32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:27.452553 19289 tablet_service.cc:1467] Processing CreateTablet for tablet c9dea4f0a48f4315bb9f9b81e4f252e9 (DEFAULT_TABLE table=test-table [id=11bbdf22f2584acbb96ef93d172be32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:27.452553 19158 tablet_service.cc:1467] Processing CreateTablet for tablet c9dea4f0a48f4315bb9f9b81e4f252e9 (DEFAULT_TABLE table=test-table [id=11bbdf22f2584acbb96ef93d172be32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:27.453527 19420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9dea4f0a48f4315bb9f9b81e4f252e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:27.453558 19289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9dea4f0a48f4315bb9f9b81e4f252e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:27.453574 19158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9dea4f0a48f4315bb9f9b81e4f252e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:27.459966 19503 tablet_bootstrap.cc:492] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: Bootstrap starting.
I20260620 14:24:27.461374 19505 tablet_bootstrap.cc:492] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: Bootstrap starting.
I20260620 14:24:27.462898 19503 tablet_bootstrap.cc:654] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:27.463101 19505 tablet_bootstrap.cc:654] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:27.463692 19503 log.cc:824] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:27.463855 19505 log.cc:824] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:27.464203 19504 tablet_bootstrap.cc:492] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357: Bootstrap starting.
I20260620 14:24:27.465648 19505 tablet_bootstrap.cc:492] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: No bootstrap required, opened a new log
I20260620 14:24:27.465778 19503 tablet_bootstrap.cc:492] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: No bootstrap required, opened a new log
I20260620 14:24:27.465888 19505 ts_tablet_manager.cc:1397] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:24:27.465965 19503 ts_tablet_manager.cc:1397] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:24:27.466456 19504 tablet_bootstrap.cc:654] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:27.467288 19504 log.cc:824] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:27.469946 19504 tablet_bootstrap.cc:492] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357: No bootstrap required, opened a new log
I20260620 14:24:27.470132 19504 ts_tablet_manager.cc:1397] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:24:27.469962 19505 raft_consensus.cc:348] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260620 14:24:27.470276 19505 raft_consensus.cc:374] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:27.470329 19505 raft_consensus.cc:729] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a6d2257ba04c1bbcd4889159636d92, State: Initialized, Role: FOLLOWER
I20260620 14:24:27.470353 19503 raft_consensus.cc:348] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260620 14:24:27.470587 19503 raft_consensus.cc:374] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:27.470638 19503 raft_consensus.cc:729] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94e3959235d4332adff4c0d4a69140a, State: Initialized, Role: FOLLOWER
I20260620 14:24:27.470806 19505 consensus_queue.cc:260] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260620 14:24:27.471069 19503 consensus_queue.cc:260] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260620 14:24:27.471726 19486 heartbeater.cc:503] Master 127.0.98.62:45561 was elected leader, sending a full tablet report...
I20260620 14:24:27.471853 19505 ts_tablet_manager.cc:1428] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:24:27.472019 19503 ts_tablet_manager.cc:1428] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:24:27.472155 19355 heartbeater.cc:503] Master 127.0.98.62:45561 was elected leader, sending a full tablet report...
I20260620 14:24:27.473825 19504 raft_consensus.cc:348] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260620 14:24:27.474094 19504 raft_consensus.cc:374] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:27.474179 19504 raft_consensus.cc:729] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591437615c4f448eb72c87ab6ac0e357, State: Initialized, Role: FOLLOWER
I20260620 14:24:27.474664 19504 consensus_queue.cc:260] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260620 14:24:27.475492 19504 ts_tablet_manager.cc:1428] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 14:24:27.475871 19224 heartbeater.cc:503] Master 127.0.98.62:45561 was elected leader, sending a full tablet report...
I20260620 14:24:27.498327 19510 raft_consensus.cc:482] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:27.498507 19510 raft_consensus.cc:504] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260620 14:24:27.499495 19510 leader_election.cc:288] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 591437615c4f448eb72c87ab6ac0e357 (127.0.98.1:40441), c1a6d2257ba04c1bbcd4889159636d92 (127.0.98.2:40433)
I20260620 14:24:27.503540 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.499854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57817 (local address 127.0.98.1:40441)
0620 14:24:27.500006 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.500011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.500025 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:24:27.500082 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.500085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.500135 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.500228 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:27.500549 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.501301 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.502448 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.503036 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.503111 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.503160 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.503377 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.503443 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:24:27.503552 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.499722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40441 (local address 127.0.98.3:57817)
0620 14:24:27.499840 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.499857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:27.499948 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:27.500263 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:27.500266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:27.500288 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:27.500408 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.500417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.501413 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.501416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.502309 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.502318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.503144 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.503148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.503247 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.503272 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:24:27.503350 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:24:27.503425 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 14:24:27.504230 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.499854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50633 (local address 127.0.98.2:40433)
0620 14:24:27.499974 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.499979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.500545 (+ 566us) server_negotiation.cc:407] Connection header received
0620 14:24:27.500741 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.500744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.500794 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.500874 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:27.501186 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.501755 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.502995 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.503571 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.503672 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.503741 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.504079 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.504141 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:24:27.504249 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.499722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40433 (local address 127.0.98.3:50633)
0620 14:24:27.500463 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.500477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:27.500609 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:27.500896 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:27.500898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:27.500914 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:27.501049 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.501056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.501867 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.501873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.502855 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.502866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.503837 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.503841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.503941 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.503963 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:24:27.504048 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:24:27.504112 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":300,"thread_start_us":146,"threads_started":1}
I20260620 14:24:27.504261 19178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dea4f0a48f4315bb9f9b81e4f252e9" candidate_uuid: "b94e3959235d4332adff4c0d4a69140a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591437615c4f448eb72c87ab6ac0e357" is_pre_election: true
I20260620 14:24:27.504607 19178 raft_consensus.cc:2454] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94e3959235d4332adff4c0d4a69140a in term 0.
I20260620 14:24:27.505055 19309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dea4f0a48f4315bb9f9b81e4f252e9" candidate_uuid: "b94e3959235d4332adff4c0d4a69140a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6d2257ba04c1bbcd4889159636d92" is_pre_election: true
I20260620 14:24:27.505165 19374 leader_election.cc:302] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591437615c4f448eb72c87ab6ac0e357, b94e3959235d4332adff4c0d4a69140a; no voters:
I20260620 14:24:27.505347 19309 raft_consensus.cc:2454] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94e3959235d4332adff4c0d4a69140a in term 0.
I20260620 14:24:27.505470 19510 raft_consensus.cc:2781] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:27.505589 19510 raft_consensus.cc:482] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:27.505652 19510 raft_consensus.cc:3037] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:27.507050 19510 raft_consensus.cc:504] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260620 14:24:27.507455 19510 leader_election.cc:288] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [CANDIDATE]: Term 1 election: Requested vote from peers 591437615c4f448eb72c87ab6ac0e357 (127.0.98.1:40441), c1a6d2257ba04c1bbcd4889159636d92 (127.0.98.2:40433)
I20260620 14:24:27.507829 19309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dea4f0a48f4315bb9f9b81e4f252e9" candidate_uuid: "b94e3959235d4332adff4c0d4a69140a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6d2257ba04c1bbcd4889159636d92"
I20260620 14:24:27.507853 19178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dea4f0a48f4315bb9f9b81e4f252e9" candidate_uuid: "b94e3959235d4332adff4c0d4a69140a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591437615c4f448eb72c87ab6ac0e357"
I20260620 14:24:27.507972 19178 raft_consensus.cc:3037] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:27.507966 19309 raft_consensus.cc:3037] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:27.509264 19178 raft_consensus.cc:2454] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94e3959235d4332adff4c0d4a69140a in term 1.
I20260620 14:24:27.509264 19309 raft_consensus.cc:2454] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94e3959235d4332adff4c0d4a69140a in term 1.
I20260620 14:24:27.509588 19374 leader_election.cc:302] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 591437615c4f448eb72c87ab6ac0e357, b94e3959235d4332adff4c0d4a69140a, c1a6d2257ba04c1bbcd4889159636d92; no voters:
I20260620 14:24:27.509744 19510 raft_consensus.cc:2781] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:27.509956 19510 raft_consensus.cc:686] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 1 LEADER]: Becoming Leader. State: Replica: b94e3959235d4332adff4c0d4a69140a, State: Running, Role: LEADER
I20260620 14:24:27.510293 19510 consensus_queue.cc:237] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260620 14:24:27.513635 19036 catalog_manager.cc:5495] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a reported cstate change: term changed from 0 to 1, leader changed from <none> to b94e3959235d4332adff4c0d4a69140a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b94e3959235d4332adff4c0d4a69140a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b94e3959235d4332adff4c0d4a69140a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } health_report { overall_health: UNKNOWN } } }
W20260620 14:24:27.529779 19225 tablet.cc:2369] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:27.530725 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.527173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.0.98.3:34271)
0620 14:24:27.527305 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.527308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.527322 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:24:27.527447 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.527450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.527506 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.527594 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:27.527845 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.528340 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.529161 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.529362 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.529438 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.529746 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:27.530270 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:27.530413 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.530516 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.530546 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:24:27.536561 19178 raft_consensus.cc:1261] T c9dea4f0a48f4315bb9f9b81e4f252e9 P 591437615c4f448eb72c87ab6ac0e357 [term 1 FOLLOWER]: Refusing update from remote peer b94e3959235d4332adff4c0d4a69140a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:27.536561 19309 raft_consensus.cc:1261] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [term 1 FOLLOWER]: Refusing update from remote peer b94e3959235d4332adff4c0d4a69140a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:27.537305 19513 consensus_queue.cc:1035] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1a6d2257ba04c1bbcd4889159636d92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:27.537487 19510 consensus_queue.cc:1035] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [LEADER]: Connected to new peer: Peer: permanent_uuid: "591437615c4f448eb72c87ab6ac0e357" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:27.554278 19034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:27.554481 19034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:27.557013 19034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47ef48c1ab8e442ea8bee0c58df16c92: Sending DeleteTablet for 3 replicas of tablet c9dea4f0a48f4315bb9f9b81e4f252e9
I20260620 14:24:27.557878 19158 tablet_service.cc:1514] Processing DeleteTablet for tablet c9dea4f0a48f4315bb9f9b81e4f252e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:27 UTC) from {username='slave'} at 127.0.0.1:42474
I20260620 14:24:27.557981 19420 tablet_service.cc:1514] Processing DeleteTablet for tablet c9dea4f0a48f4315bb9f9b81e4f252e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:27 UTC) from {username='slave'} at 127.0.0.1:54428
I20260620 14:24:27.557961 19289 tablet_service.cc:1514] Processing DeleteTablet for tablet c9dea4f0a48f4315bb9f9b81e4f252e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:27 UTC) from {username='slave'} at 127.0.0.1:46518
I20260620 14:24:27.558595 19525 tablet_replica.cc:330] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: stopping tablet replica
I20260620 14:24:27.558597 19527 tablet_replica.cc:330] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: stopping tablet replica
I20260620 14:24:27.558933 19527 raft_consensus.cc:2229] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:27.558933 19525 raft_consensus.cc:2229] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:27.559046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 19096
I20260620 14:24:27.559322 19525 raft_consensus.cc:2258] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:27.559371 19527 raft_consensus.cc:2258] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:27.561686 19525 ts_tablet_manager.cc:1905] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:27.562108 19527 ts_tablet_manager.cc:1905] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:27.564759 19527 ts_tablet_manager.cc:1918] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:27.564759 19525 ts_tablet_manager.cc:1918] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:27.564867 19525 log.cc:1196] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/c9dea4f0a48f4315bb9f9b81e4f252e9
I20260620 14:24:27.565064 19527 log.cc:1196] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/c9dea4f0a48f4315bb9f9b81e4f252e9
I20260620 14:24:27.565204 19525 ts_tablet_manager.cc:1939] T c9dea4f0a48f4315bb9f9b81e4f252e9 P b94e3959235d4332adff4c0d4a69140a: Deleting consensus metadata
I20260620 14:24:27.565510 19527 ts_tablet_manager.cc:1939] T c9dea4f0a48f4315bb9f9b81e4f252e9 P c1a6d2257ba04c1bbcd4889159636d92: Deleting consensus metadata
I20260620 14:24:27.566535 19023 catalog_manager.cc:4841] TS b94e3959235d4332adff4c0d4a69140a (127.0.98.3:34271): tablet c9dea4f0a48f4315bb9f9b81e4f252e9 (table test-table [id=11bbdf22f2584acbb96ef93d172be32d]) successfully deleted
I20260620 14:24:27.566871 19023 catalog_manager.cc:4841] TS c1a6d2257ba04c1bbcd4889159636d92 (127.0.98.2:40433): tablet c9dea4f0a48f4315bb9f9b81e4f252e9 (table test-table [id=11bbdf22f2584acbb96ef93d172be32d]) successfully deleted
W20260620 14:24:27.568535 19021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40441 (error 108)
I20260620 14:24:27.568698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 19227
I20260620 14:24:27.574846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 19358
I20260620 14:24:27.575078 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.574705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40441 (local address 127.0.0.1:42490)
0620 14:24:27.574912 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.574988 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260620 14:24:27.575587 19021 catalog_manager.cc:4584] TS 591437615c4f448eb72c87ab6ac0e357 (127.0.98.1:40441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9dea4f0a48f4315bb9f9b81e4f252e9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40441: connect: Connection refused (error 111)
I20260620 14:24:27.579922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 19005
2026-06-20T14:24:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (999 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:27 dist-test-slave-87nc krb5kdc[19531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:27 dist-test-slave-87nc krb5kdc[19531](info): set up 2 sockets
Jun 20 14:24:27 dist-test-slave-87nc krb5kdc[19531](info): commencing operation
krb5kdc: starting...
W20260620 14:24:28.643685 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:28 dist-test-slave-87nc krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:28.763114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43693
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:28.878328 19544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:28.878594 19544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:28.878634 19544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:28.882645 19544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:28.882704 19544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:28.882722 19544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:28.882737 19544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:28.882751 19544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:28.887753 19544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:28.888768 19544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:28.889757 19544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:28.898221 19552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:28.898233 19549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:28.898236 19550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:28.898516 19544 server_base.cc:1034] running on GCE node
I20260620 14:24:28.899138 19544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:28.900239 19544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:28.901450 19544 hybrid_clock.cc:648] HybridClock initialized: now 1781965468901417 us; error 44 us; skew 500 ppm
Jun 20 14:24:28 dist-test-slave-87nc krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:28.904039 19544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:28.904868 19544 webserver.cc:460] Webserver started at http://127.0.98.62:40043/ using document root <none> and password file <none>
I20260620 14:24:28.905434 19544 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:28.905543 19544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:28.905766 19544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:28.907711 19544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "9660756d116348c991341216e99588f2"
format_stamp: "Formatted at 2026-06-20 14:24:28 on dist-test-slave-87nc"
server_key: "44a80ccc62c62e8242ed67790c70feca"
server_key_iv: "e837b2edb6a6757fd5ec8b13c0e16fe3"
server_key_version: "encryptionkey@0"
I20260620 14:24:28.908339 19544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "9660756d116348c991341216e99588f2"
format_stamp: "Formatted at 2026-06-20 14:24:28 on dist-test-slave-87nc"
server_key: "44a80ccc62c62e8242ed67790c70feca"
server_key_iv: "e837b2edb6a6757fd5ec8b13c0e16fe3"
server_key_version: "encryptionkey@0"
I20260620 14:24:28.911952 19544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:28.914235 19559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:28.915402 19544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:28.915545 19544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "9660756d116348c991341216e99588f2"
format_stamp: "Formatted at 2026-06-20 14:24:28 on dist-test-slave-87nc"
server_key: "44a80ccc62c62e8242ed67790c70feca"
server_key_iv: "e837b2edb6a6757fd5ec8b13c0e16fe3"
server_key_version: "encryptionkey@0"
I20260620 14:24:28.915660 19544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:28.925647 19544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:28.928459 19544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:28.928668 19544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:28.935406 19544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43693
I20260620 14:24:28.935462 19611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43693 every 8 connection(s)
I20260620 14:24:28.936439 19544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:28.939092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 19544
I20260620 14:24:28.939224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:28.939234 19612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:28.939538 392 external_mini_cluster.cc:1428] Setting key 6e8226e648ec04a868c74d53265ad4e0
I20260620 14:24:28.946043 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2: Bootstrap starting.
Jun 20 14:24:28 dist-test-slave-87nc krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:28.948323 19612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:28.949143 19612 log.cc:824] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:28.951303 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2: No bootstrap required, opened a new log
I20260620 14:24:28.953675 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:28.940889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44474 (local address 127.0.98.62:43693)
0620 14:24:28.941311 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:28.941326 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:28.941366 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:24:28.941959 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:28.941974 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:28.942270 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:28.942538 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:28.943420 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.944432 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.945589 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.946086 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.946230 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:28.948907 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:28.948912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:28.948924 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:28.948950 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:28.950731 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:28.951218 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:28.951222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:28.951227 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:28.951332 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:28.951671 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:28.951676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:28.951682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:28.952182 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:28.952403 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:28.952697 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 14:24:28.952992 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":153,"threads_started":1}
I20260620 14:24:28.955092 19612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9660756d116348c991341216e99588f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43693 } }
I20260620 14:24:28.955350 19612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:28.955396 19612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9660756d116348c991341216e99588f2, State: Initialized, Role: FOLLOWER
I20260620 14:24:28.955883 19612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9660756d116348c991341216e99588f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43693 } }
I20260620 14:24:28.956046 19612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:28.956100 19612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:28.956171 19612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:28.957023 19612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9660756d116348c991341216e99588f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43693 } }
I20260620 14:24:28.957406 19612 leader_election.cc:302] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9660756d116348c991341216e99588f2; no voters:
I20260620 14:24:28.957759 19612 leader_election.cc:288] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:28.957901 19617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:28.958144 19617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [term 1 LEADER]: Becoming Leader. State: Replica: 9660756d116348c991341216e99588f2, State: Running, Role: LEADER
I20260620 14:24:28.958444 19617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9660756d116348c991341216e99588f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43693 } }
I20260620 14:24:28.959015 19612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:28.960074 19618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9660756d116348c991341216e99588f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9660756d116348c991341216e99588f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43693 } } }
I20260620 14:24:28.960214 19618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:28.960529 19619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9660756d116348c991341216e99588f2. Latest consensus state: current_term: 1 leader_uuid: "9660756d116348c991341216e99588f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9660756d116348c991341216e99588f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43693 } } }
I20260620 14:24:28.960624 19626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:28.960636 19619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:28.963505 19626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:28.969202 19626 catalog_manager.cc:1347] Generated new cluster ID: b347825c23f24a369fe27b8125e7b8b2
I20260620 14:24:28.969293 19626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:28.981549 19626 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:28.982615 19626 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:28.993989 19626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2: Generated new TSK 0
I20260620 14:24:28.994598 19626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:29.028569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43693
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:29.136432 19638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:29.136724 19638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:29.136801 19638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:29.140766 19638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:29.140864 19638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:29.140964 19638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:29.145985 19638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:29.147099 19638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:29.148204 19638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:29.157843 19646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:29.157853 19644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:29.158025 19638 server_base.cc:1034] running on GCE node
W20260620 14:24:29.158192 19643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:29.158641 19638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:29.159345 19638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:29.160635 19638 hybrid_clock.cc:648] HybridClock initialized: now 1781965469160607 us; error 37 us; skew 500 ppm
Jun 20 14:24:29 dist-test-slave-87nc krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:29.163596 19638 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:29.164628 19638 webserver.cc:460] Webserver started at http://127.0.98.1:43431/ using document root <none> and password file <none>
I20260620 14:24:29.165210 19638 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:29.165308 19638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:29.165547 19638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:29.167584 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "61c9b12267784d3181b60cc49bcf15ed"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-87nc"
server_key: "53c80c894633bb5d07916b672d479e49"
server_key_iv: "65173c1933971b4c2cd607ec9a7db9a2"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.168228 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "61c9b12267784d3181b60cc49bcf15ed"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-87nc"
server_key: "53c80c894633bb5d07916b672d479e49"
server_key_iv: "65173c1933971b4c2cd607ec9a7db9a2"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.171907 19638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:29.174230 19653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.175412 19638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:29.175576 19638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "61c9b12267784d3181b60cc49bcf15ed"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-87nc"
server_key: "53c80c894633bb5d07916b672d479e49"
server_key_iv: "65173c1933971b4c2cd607ec9a7db9a2"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.175693 19638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:29.199946 19638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:29.203547 19638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:29.203750 19638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:29.204399 19638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:29.205284 19638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:29.205330 19638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.205369 19638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:29.205387 19638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.214524 19638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35659
I20260620 14:24:29.214543 19766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35659 every 8 connection(s)
I20260620 14:24:29.215464 19638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:24:29 dist-test-slave-87nc krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:29.225133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 19638
I20260620 14:24:29.225274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:29.225608 392 external_mini_cluster.cc:1428] Setting key 79e226a36c1991772dbb414d076db463
I20260620 14:24:29.229336 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.217337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52705 (local address 127.0.98.62:43693)
0620 14:24:29.217648 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.217652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.218401 (+ 749us) server_negotiation.cc:407] Connection header received
0620 14:24:29.219621 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.219624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.219668 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.219732 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:29.220891 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.221407 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.222500 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.222716 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.222801 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.225257 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:29.225265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:29.225267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:29.225288 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:29.227028 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.227609 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.227611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.227613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.227684 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.228134 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.228136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.228137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.228295 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:29.228372 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.229007 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.229137 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260620 14:24:29.230054 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.217764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43693 (local address 127.0.98.1:52705)
0620 14:24:29.218233 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:29.218276 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:29.219433 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:29.219941 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:29.219948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:29.220294 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:29.220761 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.220774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.221526 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.221529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.222391 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.222399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.222858 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.222860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.222987 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.223660 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:29.223676 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:29.225087 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:29.227148 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:29.227153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:29.227166 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:29.227509 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:29.227809 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:29.227815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:29.227820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:29.228038 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:29.228399 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:29.228408 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:29.228711 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 14:24:29.228918 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 14:24:29.229185 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260620 14:24:29.231189 19767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43693
I20260620 14:24:29.231417 19767 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:29.231941 19767 heartbeater.cc:511] Master 127.0.98.62:43693 requested a full tablet report, sending...
I20260620 14:24:29.233357 19576 ts_manager.cc:194] Registered new tserver with Master: 61c9b12267784d3181b60cc49bcf15ed (127.0.98.1:35659)
I20260620 14:24:29.234445 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52705
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:29.255170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43693
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:29.365911 19772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:29.366178 19772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:29.366221 19772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:29.370326 19772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:29.370389 19772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:29.370517 19772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:29.375725 19772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:29.376746 19772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:29.377701 19772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:29.387338 19780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:29.387367 19777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:29.387400 19778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:29.387460 19772 server_base.cc:1034] running on GCE node
I20260620 14:24:29.388019 19772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:29.388634 19772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:29.389846 19772 hybrid_clock.cc:648] HybridClock initialized: now 1781965469389831 us; error 24 us; skew 500 ppm
Jun 20 14:24:29 dist-test-slave-87nc krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:29.392583 19772 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:29.393450 19772 webserver.cc:460] Webserver started at http://127.0.98.2:40155/ using document root <none> and password file <none>
I20260620 14:24:29.393996 19772 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:29.394091 19772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:29.394315 19772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:29.396351 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "5cc8ee7114034ae6aae267e72806c046"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-87nc"
server_key: "0631323b99c687832d3df09052b2c997"
server_key_iv: "2e82ba1a0b3ee78a037b19639164a15a"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.396914 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5cc8ee7114034ae6aae267e72806c046"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-87nc"
server_key: "0631323b99c687832d3df09052b2c997"
server_key_iv: "2e82ba1a0b3ee78a037b19639164a15a"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.400588 19772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:24:29.403101 19787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.404343 19772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:24:29.404479 19772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "5cc8ee7114034ae6aae267e72806c046"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-87nc"
server_key: "0631323b99c687832d3df09052b2c997"
server_key_iv: "2e82ba1a0b3ee78a037b19639164a15a"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.404595 19772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:29.430791 19772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:29.433706 19772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:29.433918 19772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:29.434500 19772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:29.435324 19772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:29.435400 19772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.435467 19772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:29.435500 19772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.444519 19772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45743
I20260620 14:24:29.444542 19900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45743 every 8 connection(s)
I20260620 14:24:29.445410 19772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:29.451964 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 19772
I20260620 14:24:29.452088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:29.452404 392 external_mini_cluster.cc:1428] Setting key 2c1b1811b3ecada90717daba7898e3bd
Jun 20 14:24:29 dist-test-slave-87nc krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:29.459878 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.447198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37205 (local address 127.0.98.62:43693)
0620 14:24:29.447366 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.447384 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.448249 (+ 865us) server_negotiation.cc:407] Connection header received
0620 14:24:29.449580 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.449584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.449644 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.449735 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:29.450915 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.451449 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.452830 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.453132 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.453235 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.455818 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:29.455822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:29.455826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:29.455857 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:29.457445 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.458030 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.458035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.458037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.458127 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.458504 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.458508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.458510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.458716 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:29.458832 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.459521 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.459679 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:24:29.460898 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.447592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43693 (local address 127.0.98.2:37205)
0620 14:24:29.448092 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:29.448133 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:29.449319 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:29.449925 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:29.449932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:29.450277 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:29.450747 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.450759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.451593 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.451597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.452695 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.452705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.453292 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.453296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.453477 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.454119 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:29.454136 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:29.455613 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:29.457602 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:29.457607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:29.457625 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:29.457906 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:29.458240 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:29.458243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:29.458245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:29.458395 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:29.458832 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:29.458840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:29.459262 (+ 422us) client_negotiation.cc:769] Sending connection context
0620 14:24:29.459469 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 14:24:29.459860 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":185,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:24:29.462311 19901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43693
I20260620 14:24:29.462540 19901 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:29.463045 19901 heartbeater.cc:511] Master 127.0.98.62:43693 requested a full tablet report, sending...
I20260620 14:24:29.464183 19576 ts_manager.cc:194] Registered new tserver with Master: 5cc8ee7114034ae6aae267e72806c046 (127.0.98.2:45743)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:24:29.464885 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:29.478744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43693
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:29.588475 19906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:29.588713 19906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:29.588754 19906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:29.592710 19906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:29.592762 19906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:29.592818 19906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:29.597937 19906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:29.598884 19906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:29.599895 19906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:29.609663 19912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:29.609663 19914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:29.609918 19906 server_base.cc:1034] running on GCE node
W20260620 14:24:29.610164 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:29.610647 19906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:29.611215 19906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:29.612378 19906 hybrid_clock.cc:648] HybridClock initialized: now 1781965469612358 us; error 30 us; skew 500 ppm
Jun 20 14:24:29 dist-test-slave-87nc krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:29.615046 19906 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:29.616012 19906 webserver.cc:460] Webserver started at http://127.0.98.3:34549/ using document root <none> and password file <none>
I20260620 14:24:29.616570 19906 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:29.616667 19906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:29.616900 19906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:29.618911 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7d2fce13aff4404a3239c7c364d2c98"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-87nc"
server_key: "6c2de6770e11cc4fe9238a7f3e68fdc8"
server_key_iv: "c4fa973989888b83937d64d5508f8305"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.619477 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7d2fce13aff4404a3239c7c364d2c98"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-87nc"
server_key: "6c2de6770e11cc4fe9238a7f3e68fdc8"
server_key_iv: "c4fa973989888b83937d64d5508f8305"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.623044 19906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:24:29.625543 19921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.626755 19906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:29.626904 19906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "d7d2fce13aff4404a3239c7c364d2c98"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-87nc"
server_key: "6c2de6770e11cc4fe9238a7f3e68fdc8"
server_key_iv: "c4fa973989888b83937d64d5508f8305"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.627022 19906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:29.664700 19906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:29.667658 19906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:29.667909 19906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:29.668507 19906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:29.669339 19906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:29.669420 19906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.669521 19906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:29.669572 19906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.678860 19906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35307
I20260620 14:24:29.678897 20034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35307 every 8 connection(s)
I20260620 14:24:29.679904 19906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:29.685328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 19906
I20260620 14:24:29.685480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:29.685784 392 external_mini_cluster.cc:1428] Setting key 4607cc5d243be665c309a0551442d7e2
Jun 20 14:24:29 dist-test-slave-87nc krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:29.693954 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.681728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36141 (local address 127.0.98.62:43693)
0620 14:24:29.681897 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.681907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.682711 (+ 804us) server_negotiation.cc:407] Connection header received
0620 14:24:29.684048 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.684052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.684105 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.684188 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:29.685482 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.686164 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.687739 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.688050 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.688134 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.690571 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:29.690573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:29.690576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:29.690598 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:29.692062 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.692577 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.692579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.692581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.692654 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.692996 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.692998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.692999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.693166 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:29.693237 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.693729 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.693826 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:24:29.694861 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.682097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43693 (local address 127.0.98.3:36141)
0620 14:24:29.682547 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:29.682589 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:29.683808 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:29.684350 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:29.684359 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:29.684701 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:29.685291 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.685314 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.686323 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.686326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.687580 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.687590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.688184 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.688187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.688324 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.688966 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:29.688982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:29.690430 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:29.692170 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:29.692176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:29.692189 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:29.692484 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:29.692735 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:29.692738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:29.692739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:29.692914 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:29.693251 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:29.693256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:29.693519 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 14:24:29.693684 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 14:24:29.693976 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260620 14:24:29.696183 20035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43693
I20260620 14:24:29.696460 20035 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:29.696997 20035 heartbeater.cc:511] Master 127.0.98.62:43693 requested a full tablet report, sending...
I20260620 14:24:29.698113 19576 ts_manager.cc:194] Registered new tserver with Master: d7d2fce13aff4404a3239c7c364d2c98 (127.0.98.3:35307)
I20260620 14:24:29.698889 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36141
I20260620 14:24:29.699729 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:29.709470 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.701946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.0.98.62:43693)
0620 14:24:29.702104 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.702110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.702294 (+ 184us) server_negotiation.cc:407] Connection header received
0620 14:24:29.702465 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.702469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.702518 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.702606 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:29.703281 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.703853 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.704691 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.704936 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.705054 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.706314 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:29.706317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:29.706320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:29.706353 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:29.707706 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.708190 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.708195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.708197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.708281 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.708585 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.708587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.708589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.708846 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:29.708959 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.709151 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.709252 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:24:29.712468 19576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:29.715406 19576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:29.729992 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.724937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.0.98.1:35659)
0620 14:24:29.725304 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.725319 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.725340 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:24:29.725412 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.725417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.725630 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.725784 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:29.725971 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.726689 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.728661 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.729226 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.729360 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.729546 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.729712 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.729813 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":72,"threads_started":1}
I20260620 14:24:29.730055 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.723717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35307 (local address 127.0.0.1:50262)
0620 14:24:29.724418 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:29.724470 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:29.724595 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:29.725309 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:29.725315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:29.725338 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:29.725497 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.725505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.727059 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.727063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.728525 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.728536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.729646 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.729650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.729738 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.729763 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:24:29.729810 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:24:29.729889 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":158,"thread_start_us":114,"threads_started":1}
I20260620 14:24:29.730329 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.724494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35659 (local address 127.0.0.1:41764)
0620 14:24:29.725101 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:29.725114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:29.725190 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:29.725768 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:29.725770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:29.725787 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:29.725878 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.725884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.726830 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.726833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.728529 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.728538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.729357 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.729361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.729471 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.729532 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 14:24:29.729682 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 14:24:29.729813 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":110,"thread_start_us":93,"threads_started":1}
I20260620 14:24:29.730991 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.723884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.0.98.3:35307)
0620 14:24:29.724765 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.724779 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.724805 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:24:29.724888 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.724893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.725123 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.725660 (+ 537us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:29.725815 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.726887 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.728669 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.729523 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.730441 (+ 918us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.730648 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.730700 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.730792 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":145,"threads_started":1}
I20260620 14:24:29.731098 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.724377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45743 (local address 127.0.0.1:39946)
0620 14:24:29.724844 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:29.724858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:29.724941 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:29.725347 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:29.725350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:29.725379 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:29.725485 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.725491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.727075 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.727077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.729406 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.729418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.730711 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.730715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.730798 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.730818 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:24:29.730888 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:24:29.730965 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":1064,"thread_start_us":100,"threads_started":1}
I20260620 14:24:29.731585 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.724054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.0.98.2:45743)
0620 14:24:29.724957 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.724968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.724987 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:24:29.725052 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.725056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.725201 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.725351 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:29.725584 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.726944 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.729595 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.730444 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.730539 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.730651 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.731352 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.731446 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":103,"threads_started":1}
I20260620 14:24:29.733528 19969 tablet_service.cc:1467] Processing CreateTablet for tablet 78b6c2af2302457286de766b7ca848e7 (DEFAULT_TABLE table=test-table [id=3e3a308728cb4bf89647fdc9b455a666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:29.733559 19701 tablet_service.cc:1467] Processing CreateTablet for tablet 78b6c2af2302457286de766b7ca848e7 (DEFAULT_TABLE table=test-table [id=3e3a308728cb4bf89647fdc9b455a666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:29.733985 19835 tablet_service.cc:1467] Processing CreateTablet for tablet 78b6c2af2302457286de766b7ca848e7 (DEFAULT_TABLE table=test-table [id=3e3a308728cb4bf89647fdc9b455a666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:29.734555 19969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b6c2af2302457286de766b7ca848e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:29.734558 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b6c2af2302457286de766b7ca848e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:29.734855 19835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b6c2af2302457286de766b7ca848e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:29.740701 20052 tablet_bootstrap.cc:492] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: Bootstrap starting.
I20260620 14:24:29.742652 20054 tablet_bootstrap.cc:492] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: Bootstrap starting.
I20260620 14:24:29.742966 20052 tablet_bootstrap.cc:654] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:29.744079 20052 log.cc:824] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:29.744102 20053 tablet_bootstrap.cc:492] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed: Bootstrap starting.
I20260620 14:24:29.744642 20054 tablet_bootstrap.cc:654] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:29.745851 20054 log.cc:824] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:29.746289 20053 tablet_bootstrap.cc:654] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:29.746294 20052 tablet_bootstrap.cc:492] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: No bootstrap required, opened a new log
I20260620 14:24:29.746529 20052 ts_tablet_manager.cc:1397] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:24:29.747104 20053 log.cc:824] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:29.747915 20054 tablet_bootstrap.cc:492] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: No bootstrap required, opened a new log
I20260620 14:24:29.748099 20054 ts_tablet_manager.cc:1397] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:24:29.749107 20053 tablet_bootstrap.cc:492] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed: No bootstrap required, opened a new log
I20260620 14:24:29.749277 20053 ts_tablet_manager.cc:1397] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:24:29.749996 20052 raft_consensus.cc:348] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } }
I20260620 14:24:29.750253 20052 raft_consensus.cc:374] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:29.750305 20052 raft_consensus.cc:729] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2fce13aff4404a3239c7c364d2c98, State: Initialized, Role: FOLLOWER
I20260620 14:24:29.750736 20052 consensus_queue.cc:260] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } }
I20260620 14:24:29.751394 20035 heartbeater.cc:503] Master 127.0.98.62:43693 was elected leader, sending a full tablet report...
I20260620 14:24:29.751705 20052 ts_tablet_manager.cc:1428] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:24:29.752662 20054 raft_consensus.cc:348] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } }
I20260620 14:24:29.752940 20054 raft_consensus.cc:374] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:29.753003 20054 raft_consensus.cc:729] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc8ee7114034ae6aae267e72806c046, State: Initialized, Role: FOLLOWER
I20260620 14:24:29.753441 20054 consensus_queue.cc:260] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } }
I20260620 14:24:29.753485 20053 raft_consensus.cc:348] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } }
I20260620 14:24:29.753762 20053 raft_consensus.cc:374] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:29.753823 20053 raft_consensus.cc:729] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c9b12267784d3181b60cc49bcf15ed, State: Initialized, Role: FOLLOWER
I20260620 14:24:29.754163 19901 heartbeater.cc:503] Master 127.0.98.62:43693 was elected leader, sending a full tablet report...
I20260620 14:24:29.754256 20053 consensus_queue.cc:260] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } }
I20260620 14:24:29.754400 20054 ts_tablet_manager.cc:1428] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:24:29.755218 19767 heartbeater.cc:503] Master 127.0.98.62:43693 was elected leader, sending a full tablet report...
I20260620 14:24:29.755285 20053 ts_tablet_manager.cc:1428] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:24:29.788698 20060 raft_consensus.cc:482] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:29.788964 20060 raft_consensus.cc:504] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } }
I20260620 14:24:29.790234 20060 leader_election.cc:288] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d2fce13aff4404a3239c7c364d2c98 (127.0.98.3:35307), 5cc8ee7114034ae6aae267e72806c046 (127.0.98.2:45743)
I20260620 14:24:29.794337 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.790588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44293 (local address 127.0.98.3:35307)
0620 14:24:29.790713 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.790718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.791056 (+ 338us) server_negotiation.cc:407] Connection header received
0620 14:24:29.791205 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.791208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.791255 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.791335 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:29.791649 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.792200 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.793318 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.793726 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.793849 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.793926 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.794150 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.794218 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:24:29.794350 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.790550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41149 (local address 127.0.98.2:45743)
0620 14:24:29.790703 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.790709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.790962 (+ 253us) server_negotiation.cc:407] Connection header received
0620 14:24:29.791150 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.791153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.791197 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.791292 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:29.791602 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.792197 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.793316 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.793731 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.793866 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.793930 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.794139 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.794224 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:24:29.794360 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.790467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35307 (local address 127.0.98.1:44293)
0620 14:24:29.790939 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:29.790951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:29.791091 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:29.791380 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:29.791383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:29.791398 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:29.791536 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.791541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.792308 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.792314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.793187 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.793196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.793851 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.793854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.794010 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.794032 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:24:29.794118 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:24:29.794205 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":248,"thread_start_us":119,"threads_started":1}
I20260620 14:24:29.794381 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.790467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45743 (local address 127.0.98.1:41149)
0620 14:24:29.790871 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:29.790886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:29.791037 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:29.791303 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:29.791305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:29.791319 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:29.791488 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.791494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.792313 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.792318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.793187 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:29.793195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.793861 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.793864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:29.793980 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.794030 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:24:29.794124 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:24:29.794239 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":66,"thread_start_us":132,"threads_started":1}
I20260620 14:24:29.795087 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b6c2af2302457286de766b7ca848e7" candidate_uuid: "61c9b12267784d3181b60cc49bcf15ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc8ee7114034ae6aae267e72806c046" is_pre_election: true
I20260620 14:24:29.795092 19989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b6c2af2302457286de766b7ca848e7" candidate_uuid: "61c9b12267784d3181b60cc49bcf15ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2fce13aff4404a3239c7c364d2c98" is_pre_election: true
I20260620 14:24:29.795394 19855 raft_consensus.cc:2454] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61c9b12267784d3181b60cc49bcf15ed in term 0.
I20260620 14:24:29.795423 19989 raft_consensus.cc:2454] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61c9b12267784d3181b60cc49bcf15ed in term 0.
I20260620 14:24:29.795928 19657 leader_election.cc:302] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc8ee7114034ae6aae267e72806c046, 61c9b12267784d3181b60cc49bcf15ed; no voters:
I20260620 14:24:29.796205 20060 raft_consensus.cc:2781] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:29.796291 20060 raft_consensus.cc:482] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:29.796334 20060 raft_consensus.cc:3037] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:29.797312 20060 raft_consensus.cc:504] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } }
I20260620 14:24:29.797622 20060 leader_election.cc:288] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [CANDIDATE]: Term 1 election: Requested vote from peers d7d2fce13aff4404a3239c7c364d2c98 (127.0.98.3:35307), 5cc8ee7114034ae6aae267e72806c046 (127.0.98.2:45743)
I20260620 14:24:29.797940 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b6c2af2302457286de766b7ca848e7" candidate_uuid: "61c9b12267784d3181b60cc49bcf15ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc8ee7114034ae6aae267e72806c046"
I20260620 14:24:29.797940 19989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b6c2af2302457286de766b7ca848e7" candidate_uuid: "61c9b12267784d3181b60cc49bcf15ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2fce13aff4404a3239c7c364d2c98"
I20260620 14:24:29.798053 19855 raft_consensus.cc:3037] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:29.798053 19989 raft_consensus.cc:3037] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:29.799249 19989 raft_consensus.cc:2454] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61c9b12267784d3181b60cc49bcf15ed in term 1.
I20260620 14:24:29.799249 19855 raft_consensus.cc:2454] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61c9b12267784d3181b60cc49bcf15ed in term 1.
I20260620 14:24:29.799541 19656 leader_election.cc:302] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c9b12267784d3181b60cc49bcf15ed, d7d2fce13aff4404a3239c7c364d2c98; no voters:
I20260620 14:24:29.799690 20060 raft_consensus.cc:2781] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:29.799980 20060 raft_consensus.cc:686] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [term 1 LEADER]: Becoming Leader. State: Replica: 61c9b12267784d3181b60cc49bcf15ed, State: Running, Role: LEADER
I20260620 14:24:29.800329 20060 consensus_queue.cc:237] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } }
I20260620 14:24:29.803419 19575 catalog_manager.cc:5495] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c9b12267784d3181b60cc49bcf15ed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "61c9b12267784d3181b60cc49bcf15ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61c9b12267784d3181b60cc49bcf15ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } health_report { overall_health: HEALTHY } } }
I20260620 14:24:29.828358 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.824716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.0.98.1:35659)
0620 14:24:29.824889 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.824892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.824907 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:24:29.825013 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.825016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.825069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.825155 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:29.825369 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.825891 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.826749 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.827007 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.827093 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.827374 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:29.827932 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:29.828030 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.828116 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.828169 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 14:24:29.834604 19989 raft_consensus.cc:1261] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [term 1 FOLLOWER]: Refusing update from remote peer 61c9b12267784d3181b60cc49bcf15ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:29.834607 19855 raft_consensus.cc:1261] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [term 1 FOLLOWER]: Refusing update from remote peer 61c9b12267784d3181b60cc49bcf15ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:29.835350 20063 consensus_queue.cc:1035] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2fce13aff4404a3239c7c364d2c98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:29.835569 20060 consensus_queue.cc:1035] T 78b6c2af2302457286de766b7ca848e7 P 61c9b12267784d3181b60cc49bcf15ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc8ee7114034ae6aae267e72806c046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:29.842712 20066 mvcc.cc:205] Tried to move back new op lower bound from 7298930564435316736 to 7298930564305281024. Current Snapshot: MvccSnapshot[applied={T|T < 7298930564435316736}]
I20260620 14:24:29.844901 20068 mvcc.cc:205] Tried to move back new op lower bound from 7298930564435316736 to 7298930564305281024. Current Snapshot: MvccSnapshot[applied={T|T < 7298930564435316736}]
I20260620 14:24:29.852653 19576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:29.852845 19576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:29.855340 19576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9660756d116348c991341216e99588f2: Sending DeleteTablet for 3 replicas of tablet 78b6c2af2302457286de766b7ca848e7
I20260620 14:24:29.856208 19835 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b6c2af2302457286de766b7ca848e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39946
I20260620 14:24:29.856376 19969 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b6c2af2302457286de766b7ca848e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50262
I20260620 14:24:29.856782 20075 tablet_replica.cc:330] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: stopping tablet replica
I20260620 14:24:29.856848 19701 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b6c2af2302457286de766b7ca848e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41764
I20260620 14:24:29.857214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 19638
I20260620 14:24:29.857332 20076 tablet_replica.cc:330] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: stopping tablet replica
I20260620 14:24:29.857667 20076 raft_consensus.cc:2229] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:29.857919 20075 raft_consensus.cc:2229] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:29.857992 20076 raft_consensus.cc:2258] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:29.858222 20075 raft_consensus.cc:2258] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:29.860565 20075 ts_tablet_manager.cc:1905] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:29.860790 20076 ts_tablet_manager.cc:1905] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:29.862881 20075 ts_tablet_manager.cc:1918] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:29.863003 20075 log.cc:1196] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/78b6c2af2302457286de766b7ca848e7
I20260620 14:24:29.863291 20075 ts_tablet_manager.cc:1939] T 78b6c2af2302457286de766b7ca848e7 P 5cc8ee7114034ae6aae267e72806c046: Deleting consensus metadata
I20260620 14:24:29.864167 20076 ts_tablet_manager.cc:1918] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:29.864269 20076 log.cc:1196] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/78b6c2af2302457286de766b7ca848e7
I20260620 14:24:29.864615 20076 ts_tablet_manager.cc:1939] T 78b6c2af2302457286de766b7ca848e7 P d7d2fce13aff4404a3239c7c364d2c98: Deleting consensus metadata
I20260620 14:24:29.864604 19563 catalog_manager.cc:4841] TS 5cc8ee7114034ae6aae267e72806c046 (127.0.98.2:45743): tablet 78b6c2af2302457286de766b7ca848e7 (table test-table [id=3e3a308728cb4bf89647fdc9b455a666]) successfully deleted
W20260620 14:24:29.865566 19562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35659 (error 108)
I20260620 14:24:29.866019 19562 catalog_manager.cc:4841] TS d7d2fce13aff4404a3239c7c364d2c98 (127.0.98.3:35307): tablet 78b6c2af2302457286de766b7ca848e7 (table test-table [id=3e3a308728cb4bf89647fdc9b455a666]) successfully deleted
I20260620 14:24:29.866537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 19772
I20260620 14:24:29.866639 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.866327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35659 (local address 127.0.0.1:41768)
0620 14:24:29.866470 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:29.866532 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:24:29.867491 19562 catalog_manager.cc:4584] TS 61c9b12267784d3181b60cc49bcf15ed (127.0.98.1:35659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78b6c2af2302457286de766b7ca848e7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35659: connect: Connection refused (error 111)
I20260620 14:24:29.872506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 19906
I20260620 14:24:29.878459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 19544
2026-06-20T14:24:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2299 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:29Z Disabled control of system clock
I20260620 14:24:29.908504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43213
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:24:30.018198 20082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:30.018498 20082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:30.018536 20082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:30.022534 20082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:30.022601 20082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:30.022616 20082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:30.022634 20082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:30.022648 20082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:30.027426 20082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:30.028501 20082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:30.029877 20082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:30.038446 20082 server_base.cc:1034] running on GCE node
W20260620 14:24:30.038455 20088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:30.038436 20087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:30.038414 20090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:30.039287 20082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:30.040494 20082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:30.041697 20082 hybrid_clock.cc:648] HybridClock initialized: now 1781965470041660 us; error 53 us; skew 500 ppm
I20260620 14:24:30.043507 20082 webserver.cc:460] Webserver started at http://127.0.98.62:35677/ using document root <none> and password file <none>
I20260620 14:24:30.044147 20082 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:30.044245 20082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:30.044483 20082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:30.046516 20082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "6db57b5a70e24110aa83134156d3fd28"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "790577fb048445be43e78a7f7bfead09"
server_key_iv: "06240ac58e1efc1614a514e23b4fbc98"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.047092 20082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "6db57b5a70e24110aa83134156d3fd28"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "790577fb048445be43e78a7f7bfead09"
server_key_iv: "06240ac58e1efc1614a514e23b4fbc98"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.051061 20082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:24:30.053571 20096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.054879 20082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:24:30.054977 20082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "6db57b5a70e24110aa83134156d3fd28"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "790577fb048445be43e78a7f7bfead09"
server_key_iv: "06240ac58e1efc1614a514e23b4fbc98"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.055064 20082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:30.072741 20082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:30.073700 20082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:30.073911 20082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:30.082607 20082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43213
I20260620 14:24:30.082610 20148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43213 every 8 connection(s)
I20260620 14:24:30.083591 20082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:30.084018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 20082
I20260620 14:24:30.084162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:30.084475 392 external_mini_cluster.cc:1428] Setting key 532f5dd12eae6f9469cda05551d48723
I20260620 14:24:30.087111 20149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:30.092687 20149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28: Bootstrap starting.
I20260620 14:24:30.093932 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.085920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.0.98.62:43213)
0620 14:24:30.086549 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.086563 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.086599 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:24:30.087352 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.087371 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.087758 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.088186 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:30.088643 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.089654 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.090566 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.091047 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.091236 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.092216 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:30.092221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:30.092234 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:30.092619 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:30.092673 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:30.092692 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:30.092811 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.092960 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.093214 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260620 14:24:30.095103 20149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:30.096004 20149 log.cc:824] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:30.097900 20149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28: No bootstrap required, opened a new log
I20260620 14:24:30.101583 20149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db57b5a70e24110aa83134156d3fd28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43213 } }
I20260620 14:24:30.101786 20149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:30.101824 20149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db57b5a70e24110aa83134156d3fd28, State: Initialized, Role: FOLLOWER
I20260620 14:24:30.102248 20149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db57b5a70e24110aa83134156d3fd28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43213 } }
I20260620 14:24:30.102376 20149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:30.102414 20149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:30.102515 20149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:30.103323 20149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db57b5a70e24110aa83134156d3fd28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43213 } }
I20260620 14:24:30.103629 20149 leader_election.cc:302] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6db57b5a70e24110aa83134156d3fd28; no voters:
I20260620 14:24:30.104009 20149 leader_election.cc:288] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:30.104120 20154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:30.104372 20154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [term 1 LEADER]: Becoming Leader. State: Replica: 6db57b5a70e24110aa83134156d3fd28, State: Running, Role: LEADER
I20260620 14:24:30.104665 20154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db57b5a70e24110aa83134156d3fd28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43213 } }
I20260620 14:24:30.105221 20149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:30.106216 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6db57b5a70e24110aa83134156d3fd28. Latest consensus state: current_term: 1 leader_uuid: "6db57b5a70e24110aa83134156d3fd28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db57b5a70e24110aa83134156d3fd28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43213 } } }
I20260620 14:24:30.106206 20156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6db57b5a70e24110aa83134156d3fd28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db57b5a70e24110aa83134156d3fd28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43213 } } }
I20260620 14:24:30.106353 20156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:30.106354 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:30.106709 20163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:30.109905 20163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:30.116009 20163 catalog_manager.cc:1347] Generated new cluster ID: ee88fbede1ea49deba54bb1a916f50ff
I20260620 14:24:30.116076 20163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:30.141060 20163 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:30.142316 20163 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:30.147514 20163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28: Generated new TSK 0
I20260620 14:24:30.148278 20163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:30.158084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43213
--builtin_ntp_servers=127.0.98.20:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:24:30.270449 20173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:30.270736 20173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:30.270776 20173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:30.274916 20173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:30.274993 20173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:30.275098 20173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:30.280042 20173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:30.281185 20173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:30.282629 20173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:30.292189 20178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:30.292263 20173 server_base.cc:1034] running on GCE node
W20260620 14:24:30.292196 20181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:30.292184 20179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:30.292804 20173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:30.293404 20173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:30.294574 20173 hybrid_clock.cc:648] HybridClock initialized: now 1781965470294561 us; error 24 us; skew 500 ppm
I20260620 14:24:30.296306 20173 webserver.cc:460] Webserver started at http://127.0.98.1:36957/ using document root <none> and password file <none>
I20260620 14:24:30.296831 20173 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:30.296888 20173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:30.297060 20173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:30.298976 20173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/data/instance:
uuid: "0e24e9dd8d224fbdacd3d65815803739"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "a8ce9224a6b8dc16371b337015832447"
server_key_iv: "afff3cd924448943fde29db89379b602"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.299463 20173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0e24e9dd8d224fbdacd3d65815803739"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "a8ce9224a6b8dc16371b337015832447"
server_key_iv: "afff3cd924448943fde29db89379b602"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.303517 20173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 14:24:30.306221 20187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.307653 20173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:24:30.307746 20173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/wal
uuid: "0e24e9dd8d224fbdacd3d65815803739"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "a8ce9224a6b8dc16371b337015832447"
server_key_iv: "afff3cd924448943fde29db89379b602"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.307893 20173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:30.334724 20173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:30.335404 20173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:30.335551 20173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:30.336097 20173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:30.336959 20173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:30.337005 20173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.337044 20173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:30.337059 20173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.345563 20173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39597
I20260620 14:24:30.345605 20300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39597 every 8 connection(s)
I20260620 14:24:30.346437 20173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:30.355050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 20173
I20260620 14:24:30.355196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:30.355569 392 external_mini_cluster.cc:1428] Setting key 82e4b80e8c92f63c1d31195a3fa90e6d
I20260620 14:24:30.355727 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.348207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48799 (local address 127.0.98.62:43213)
0620 14:24:30.348356 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.348360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.349493 (+ 1133us) server_negotiation.cc:407] Connection header received
0620 14:24:30.350871 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.350875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.350935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.351023 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:30.351944 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.352497 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.353647 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.353887 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.354006 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.355105 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:30.355109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:30.355112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:30.355125 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:30.355153 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:30.355161 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:30.355256 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.355483 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.355557 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:24:30.356689 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.348549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43213 (local address 127.0.98.1:48799)
0620 14:24:30.349314 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:30.349357 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:30.350653 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:30.351216 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:30.351222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:30.351303 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:30.351785 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.351798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.352631 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.352634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.353508 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.353516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.354043 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.354047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.354211 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.354850 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:30.354851 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:30.354889 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:30.354947 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:30.355291 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:30.355300 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:30.355308 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:24:30.355475 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:24:30.355754 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260620 14:24:30.357705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43213
--builtin_ntp_servers=127.0.98.20:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:24:30.358022 20301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43213
I20260620 14:24:30.358263 20301 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:30.358821 20301 heartbeater.cc:511] Master 127.0.98.62:43213 requested a full tablet report, sending...
I20260620 14:24:30.360518 20113 ts_manager.cc:194] Registered new tserver with Master: 0e24e9dd8d224fbdacd3d65815803739 (127.0.98.1:39597)
I20260620 14:24:30.361728 20113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48799
W20260620 14:24:30.470456 20304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:30.470793 20304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:30.470868 20304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:30.475122 20304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:30.475219 20304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:30.475311 20304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:30.480283 20304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:30.481451 20304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:30.482846 20304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:30.492738 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:30.492790 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:30.492973 20304 server_base.cc:1034] running on GCE node
W20260620 14:24:30.493073 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:30.493604 20304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:30.494477 20304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:30.495714 20304 hybrid_clock.cc:648] HybridClock initialized: now 1781965470495687 us; error 36 us; skew 500 ppm
I20260620 14:24:30.498310 20304 webserver.cc:460] Webserver started at http://127.0.98.2:39395/ using document root <none> and password file <none>
I20260620 14:24:30.499095 20304 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:30.499202 20304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:30.499480 20304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:30.502259 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/data/instance:
uuid: "68b6092d49304d438c259eabd100128c"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "62e3a2d436d027c062abe0e3c8bb1f56"
server_key_iv: "63215a1f5699c9121e6b0cbf5dc26e0b"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.502784 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/wal/instance:
uuid: "68b6092d49304d438c259eabd100128c"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "62e3a2d436d027c062abe0e3c8bb1f56"
server_key_iv: "63215a1f5699c9121e6b0cbf5dc26e0b"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.507110 20304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:24:30.510010 20318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.511384 20304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:24:30.511523 20304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/wal
uuid: "68b6092d49304d438c259eabd100128c"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "62e3a2d436d027c062abe0e3c8bb1f56"
server_key_iv: "63215a1f5699c9121e6b0cbf5dc26e0b"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.511652 20304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:30.525957 20304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:30.526705 20304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:30.526922 20304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:30.527545 20304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:30.528645 20304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:30.528723 20304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.528788 20304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:30.528820 20304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.537776 20304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33851
I20260620 14:24:30.537809 20431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33851 every 8 connection(s)
I20260620 14:24:30.538719 20304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:30.544409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 20304
I20260620 14:24:30.544551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:30.544879 392 external_mini_cluster.cc:1428] Setting key 48c988fe1cfa0dea4881cac9e291357c
I20260620 14:24:30.547035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43213
--builtin_ntp_servers=127.0.98.20:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:24:30.549150 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.540764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48377 (local address 127.0.98.62:43213)
0620 14:24:30.540896 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.540901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.542038 (+ 1137us) server_negotiation.cc:407] Connection header received
0620 14:24:30.543383 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.543386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.543439 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.543523 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:30.544767 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.545424 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.546916 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.547216 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.547360 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.548688 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:30.548691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:30.548693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:30.548705 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:30.548724 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:30.548732 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:30.548792 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.548987 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.549042 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:24:30.550442 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.541163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43213 (local address 127.0.98.2:48377)
0620 14:24:30.541847 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:30.541896 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:30.543141 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:30.543689 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:30.543695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:30.543853 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:30.544565 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.544588 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.545561 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.545564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.546750 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.546761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.547406 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.547410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.547588 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.548340 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:30.548357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:30.548425 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:30.548518 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:30.548822 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:30.548834 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:30.548840 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:30.548979 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:24:30.549310 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260620 14:24:30.551681 20432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43213
I20260620 14:24:30.551954 20432 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:30.552475 20432 heartbeater.cc:511] Master 127.0.98.62:43213 requested a full tablet report, sending...
I20260620 14:24:30.553546 20113 ts_manager.cc:194] Registered new tserver with Master: 68b6092d49304d438c259eabd100128c (127.0.98.2:33851)
I20260620 14:24:30.554131 20113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48377
W20260620 14:24:30.662058 20435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:30.662406 20435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:30.662487 20435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:30.666658 20435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:30.666777 20435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:30.666887 20435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:30.671820 20435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:30.673056 20435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:30.674438 20435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:30.684442 20435 server_base.cc:1034] running on GCE node
W20260620 14:24:30.684401 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:30.684382 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:30.684407 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:30.685248 20435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:30.686029 20435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:30.687252 20435 hybrid_clock.cc:648] HybridClock initialized: now 1781965470687234 us; error 33 us; skew 500 ppm
I20260620 14:24:30.689370 20435 webserver.cc:460] Webserver started at http://127.0.98.3:38299/ using document root <none> and password file <none>
I20260620 14:24:30.690016 20435 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:30.690119 20435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:30.690357 20435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:30.692523 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/data/instance:
uuid: "2fd47bb10d5b45f48570bfad2fa3dda5"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "6536709289a13e3134276beccdd428f3"
server_key_iv: "3fce743a9f315796e4734b4f7ffe8ffa"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.693109 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2fd47bb10d5b45f48570bfad2fa3dda5"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "6536709289a13e3134276beccdd428f3"
server_key_iv: "3fce743a9f315796e4734b4f7ffe8ffa"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.696942 20435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:24:30.699447 20449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.700794 20435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:30.700969 20435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/wal
uuid: "2fd47bb10d5b45f48570bfad2fa3dda5"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-87nc"
server_key: "6536709289a13e3134276beccdd428f3"
server_key_iv: "3fce743a9f315796e4734b4f7ffe8ffa"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.701092 20435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:30.727859 20435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:30.728612 20435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:30.728825 20435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:30.729431 20435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:30.730415 20435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:30.730490 20435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.730557 20435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:30.730609 20435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.740552 20435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39933
I20260620 14:24:30.740608 20562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39933 every 8 connection(s)
I20260620 14:24:30.741501 20435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:30.744416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 20435
I20260620 14:24:30.744530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:30.744848 392 external_mini_cluster.cc:1428] Setting key 4f1c5ab8a38b141b1e0d41c6e7fe02d9
I20260620 14:24:30.751329 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.743347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41451 (local address 127.0.98.62:43213)
0620 14:24:30.743498 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.743502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.744815 (+ 1313us) server_negotiation.cc:407] Connection header received
0620 14:24:30.746448 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.746452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.746507 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.746594 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:30.747599 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.748206 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.749362 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.749597 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.749719 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.750818 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:30.750822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:30.750825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:30.750844 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:30.750865 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:30.750873 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:30.750961 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.751118 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.751178 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:24:30.752202 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.743746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43213 (local address 127.0.98.3:41451)
0620 14:24:30.744635 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:30.744694 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:30.746221 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:30.746780 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:30.746787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:30.746879 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:30.747462 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.747475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.748337 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.748341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.749224 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.749232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.749753 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.749757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.749993 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.750559 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:30.750560 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:30.750599 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:30.750659 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:30.750963 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:30.750974 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:30.750982 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:24:30.751118 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 14:24:30.751385 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":105,"threads_started":1}
I20260620 14:24:30.753314 20563 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43213
I20260620 14:24:30.753516 20563 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:30.753968 20563 heartbeater.cc:511] Master 127.0.98.62:43213 requested a full tablet report, sending...
I20260620 14:24:30.754877 20113 ts_manager.cc:194] Registered new tserver with Master: 2fd47bb10d5b45f48570bfad2fa3dda5 (127.0.98.3:39933)
I20260620 14:24:30.755496 20113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41451
I20260620 14:24:30.759176 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:30.764972 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.761157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.0.98.62:43213)
0620 14:24:30.761318 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.761323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.761472 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:24:30.761595 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.761599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.761642 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.761738 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:30.762040 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.762550 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.763412 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.763654 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.763802 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.764540 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:30.764544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:30.764547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:30.764577 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:30.764601 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:30.764609 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:30.764695 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.764782 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.764832 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:24:30.767437 20113 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:30.770367 20113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:30.784927 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.778494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33851 (local address 127.0.0.1:42202)
0620 14:24:30.779555 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:30.779609 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:30.779751 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:30.780366 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:30.780372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:30.780394 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:30.780563 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.780571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.782131 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.782134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.783404 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.783414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.784360 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.784364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.784478 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.784515 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:24:30.784627 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:24:30.784753 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":57,"spinlock_wait_cycles":3328,"thread_start_us":76,"threads_started":1}
I20260620 14:24:30.784929 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.778613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.0.98.2:33851)
0620 14:24:30.779035 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.779048 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.779942 (+ 894us) server_negotiation.cc:407] Connection header received
0620 14:24:30.780031 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.780036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.780195 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.780369 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:30.780720 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.781911 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.783552 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.784228 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.784371 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.784576 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.784657 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.784753 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":121,"threads_started":1}
I20260620 14:24:30.785384 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.778567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39597 (local address 127.0.0.1:47090)
0620 14:24:30.779554 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:30.779609 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:30.779750 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:30.780450 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:30.780452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:30.780466 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:30.780616 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.780621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.781874 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.781877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.783406 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.783415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.784778 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.784782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.784916 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.784967 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:24:30.785173 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 14:24:30.785243 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":149,"spinlock_wait_cycles":4992,"thread_start_us":103,"threads_started":1}
I20260620 14:24:30.785862 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.779232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39933 (local address 127.0.0.1:32814)
0620 14:24:30.780595 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:30.780610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:30.780694 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:30.781241 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:30.781244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:30.781263 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:30.781346 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.781352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.782700 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.782703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.783535 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.783546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.784778 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.784782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.784911 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.784956 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:24:30.785029 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:24:30.785737 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":32,"thread_start_us":99,"threads_started":1}
I20260620 14:24:30.786037 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.778971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.0.98.1:39597)
0620 14:24:30.779242 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.779253 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.779948 (+ 695us) server_negotiation.cc:407] Connection header received
0620 14:24:30.780030 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.780034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.780194 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.780313 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:30.780793 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.781749 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.783551 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.784428 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.784536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.784711 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.785656 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.785755 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260620 14:24:30.786379 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.779413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.0.98.3:39933)
0620 14:24:30.779888 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.779902 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.780793 (+ 891us) server_negotiation.cc:407] Connection header received
0620 14:24:30.780900 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.780905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.781078 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.781221 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:30.781472 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.782577 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.783659 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.784428 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.784536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.784711 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.785309 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.785430 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":161,"threads_started":1}
I20260620 14:24:30.788137 20366 tablet_service.cc:1467] Processing CreateTablet for tablet a986dbb589b54212b390d5bdc3a92893 (DEFAULT_TABLE table=test-table [id=fe18a3b8685144e6a2f8aaf9c45d4e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:30.789153 20366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a986dbb589b54212b390d5bdc3a92893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:30.788772 20235 tablet_service.cc:1467] Processing CreateTablet for tablet a986dbb589b54212b390d5bdc3a92893 (DEFAULT_TABLE table=test-table [id=fe18a3b8685144e6a2f8aaf9c45d4e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:30.788978 20497 tablet_service.cc:1467] Processing CreateTablet for tablet a986dbb589b54212b390d5bdc3a92893 (DEFAULT_TABLE table=test-table [id=fe18a3b8685144e6a2f8aaf9c45d4e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:30.789883 20235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a986dbb589b54212b390d5bdc3a92893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:30.789882 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a986dbb589b54212b390d5bdc3a92893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:30.795683 20580 tablet_bootstrap.cc:492] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: Bootstrap starting.
I20260620 14:24:30.796876 20581 tablet_bootstrap.cc:492] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: Bootstrap starting.
I20260620 14:24:30.798147 20582 tablet_bootstrap.cc:492] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739: Bootstrap starting.
I20260620 14:24:30.798504 20580 tablet_bootstrap.cc:654] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:30.799008 20581 tablet_bootstrap.cc:654] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:30.799407 20580 log.cc:824] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:30.799841 20581 log.cc:824] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:30.800177 20582 tablet_bootstrap.cc:654] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:30.800958 20582 log.cc:824] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:30.801872 20580 tablet_bootstrap.cc:492] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: No bootstrap required, opened a new log
I20260620 14:24:30.801864 20581 tablet_bootstrap.cc:492] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: No bootstrap required, opened a new log
I20260620 14:24:30.802059 20580 ts_tablet_manager.cc:1397] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260620 14:24:30.802059 20581 ts_tablet_manager.cc:1397] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:24:30.802815 20582 tablet_bootstrap.cc:492] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739: No bootstrap required, opened a new log
I20260620 14:24:30.803022 20582 ts_tablet_manager.cc:1397] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:24:30.806509 20580 raft_consensus.cc:348] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } }
I20260620 14:24:30.806509 20581 raft_consensus.cc:348] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } }
I20260620 14:24:30.806784 20581 raft_consensus.cc:374] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:30.806849 20581 raft_consensus.cc:729] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd47bb10d5b45f48570bfad2fa3dda5, State: Initialized, Role: FOLLOWER
I20260620 14:24:30.806784 20580 raft_consensus.cc:374] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:30.807054 20580 raft_consensus.cc:729] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b6092d49304d438c259eabd100128c, State: Initialized, Role: FOLLOWER
I20260620 14:24:30.806967 20582 raft_consensus.cc:348] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } }
I20260620 14:24:30.807221 20582 raft_consensus.cc:374] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:30.807271 20582 raft_consensus.cc:729] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e24e9dd8d224fbdacd3d65815803739, State: Initialized, Role: FOLLOWER
I20260620 14:24:30.807397 20581 consensus_queue.cc:260] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } }
I20260620 14:24:30.807699 20582 consensus_queue.cc:260] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } }
I20260620 14:24:30.807937 20580 consensus_queue.cc:260] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } }
I20260620 14:24:30.808447 20581 ts_tablet_manager.cc:1428] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:24:30.808497 20301 heartbeater.cc:503] Master 127.0.98.62:43213 was elected leader, sending a full tablet report...
I20260620 14:24:30.808692 20582 ts_tablet_manager.cc:1428] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:24:30.808765 20563 heartbeater.cc:503] Master 127.0.98.62:43213 was elected leader, sending a full tablet report...
I20260620 14:24:30.808812 20580 ts_tablet_manager.cc:1428] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260620 14:24:30.809163 20432 heartbeater.cc:503] Master 127.0.98.62:43213 was elected leader, sending a full tablet report...
I20260620 14:24:30.813854 20587 raft_consensus.cc:482] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:30.814003 20587 raft_consensus.cc:504] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } }
I20260620 14:24:30.815054 20587 leader_election.cc:288] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68b6092d49304d438c259eabd100128c (127.0.98.2:33851), 2fd47bb10d5b45f48570bfad2fa3dda5 (127.0.98.3:39933)
I20260620 14:24:30.819458 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.815235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33851 (local address 127.0.98.1:34121)
0620 14:24:30.815380 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:30.815412 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:30.815502 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:30.815830 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:30.815833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:30.815849 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:30.815997 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.816005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.817024 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.817027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.818094 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.818103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.819028 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.819032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.819130 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.819152 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:24:30.819254 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:24:30.819328 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 14:24:30.819458 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.815295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34121 (local address 127.0.98.2:33851)
0620 14:24:30.815397 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.815403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.815456 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:24:30.815638 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.815641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.815688 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.815753 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:30.816196 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.816827 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.818222 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.818906 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.819024 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.819083 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.819268 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.819337 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:24:30.820149 20386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a986dbb589b54212b390d5bdc3a92893" candidate_uuid: "0e24e9dd8d224fbdacd3d65815803739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b6092d49304d438c259eabd100128c" is_pre_election: true
I20260620 14:24:30.820235 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.816178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45711 (local address 127.0.98.3:39933)
0620 14:24:30.816284 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.816288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.816302 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:24:30.816359 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.816362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.816401 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.816475 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:30.816874 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.817572 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.819053 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.819489 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.819596 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.819666 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.820061 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.820132 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:24:30.820250 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.815804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39933 (local address 127.0.98.1:45711)
0620 14:24:30.816132 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:30.816145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:30.816219 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:30.816580 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:30.816583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:30.816599 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:30.816726 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.816731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.817718 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.817721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.818916 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:30.818927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.819814 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.819818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:30.819919 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.819942 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:24:30.820046 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:24:30.820114 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260620 14:24:30.820490 20386 raft_consensus.cc:2454] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e24e9dd8d224fbdacd3d65815803739 in term 0.
I20260620 14:24:30.820885 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a986dbb589b54212b390d5bdc3a92893" candidate_uuid: "0e24e9dd8d224fbdacd3d65815803739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" is_pre_election: true
I20260620 14:24:30.821019 20188 leader_election.cc:302] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e24e9dd8d224fbdacd3d65815803739, 68b6092d49304d438c259eabd100128c; no voters:
I20260620 14:24:30.821183 20517 raft_consensus.cc:2454] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e24e9dd8d224fbdacd3d65815803739 in term 0.
I20260620 14:24:30.821333 20587 raft_consensus.cc:2781] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:30.821430 20587 raft_consensus.cc:482] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:30.821493 20587 raft_consensus.cc:3037] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:30.822528 20587 raft_consensus.cc:504] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } }
I20260620 14:24:30.822813 20587 leader_election.cc:288] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [CANDIDATE]: Term 1 election: Requested vote from peers 68b6092d49304d438c259eabd100128c (127.0.98.2:33851), 2fd47bb10d5b45f48570bfad2fa3dda5 (127.0.98.3:39933)
I20260620 14:24:30.823232 20386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a986dbb589b54212b390d5bdc3a92893" candidate_uuid: "0e24e9dd8d224fbdacd3d65815803739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b6092d49304d438c259eabd100128c"
I20260620 14:24:30.823233 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a986dbb589b54212b390d5bdc3a92893" candidate_uuid: "0e24e9dd8d224fbdacd3d65815803739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5"
I20260620 14:24:30.823336 20386 raft_consensus.cc:3037] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:30.823364 20517 raft_consensus.cc:3037] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:30.824386 20386 raft_consensus.cc:2454] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e24e9dd8d224fbdacd3d65815803739 in term 1.
I20260620 14:24:30.824532 20517 raft_consensus.cc:2454] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e24e9dd8d224fbdacd3d65815803739 in term 1.
I20260620 14:24:30.824726 20188 leader_election.cc:302] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e24e9dd8d224fbdacd3d65815803739, 68b6092d49304d438c259eabd100128c; no voters:
I20260620 14:24:30.824898 20587 raft_consensus.cc:2781] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:30.825134 20587 raft_consensus.cc:686] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 1 LEADER]: Becoming Leader. State: Replica: 0e24e9dd8d224fbdacd3d65815803739, State: Running, Role: LEADER
I20260620 14:24:30.825469 20587 consensus_queue.cc:237] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } }
I20260620 14:24:30.828718 20111 catalog_manager.cc:5495] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e24e9dd8d224fbdacd3d65815803739 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0e24e9dd8d224fbdacd3d65815803739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e24e9dd8d224fbdacd3d65815803739" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 } health_report { overall_health: UNKNOWN } } }
W20260620 14:24:30.847551 20302 tablet.cc:2369] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:30.850641 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.846919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.0.98.1:39597)
0620 14:24:30.847043 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.847046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.847059 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:24:30.847212 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.847214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.847270 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.847378 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:30.847712 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.848289 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.849154 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.849359 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.849436 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.849723 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:30.850252 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:30.850397 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.850487 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.850537 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:24:30.856702 20386 raft_consensus.cc:1261] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [term 1 FOLLOWER]: Refusing update from remote peer 0e24e9dd8d224fbdacd3d65815803739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:30.856667 20517 raft_consensus.cc:1261] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [term 1 FOLLOWER]: Refusing update from remote peer 0e24e9dd8d224fbdacd3d65815803739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:30.857470 20590 consensus_queue.cc:1035] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd47bb10d5b45f48570bfad2fa3dda5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:30.857637 20587 consensus_queue.cc:1035] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b6092d49304d438c259eabd100128c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:30.866437 20595 mvcc.cc:205] Tried to move back new op lower bound from 7298930568621268992 to 7298930568504168448. Current Snapshot: MvccSnapshot[applied={T|T < 7298930568621268992}]
I20260620 14:24:30.867413 20594 mvcc.cc:205] Tried to move back new op lower bound from 7298930568621268992 to 7298930568504168448. Current Snapshot: MvccSnapshot[applied={T|T < 7298930568621268992}]
I20260620 14:24:30.875452 20112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:30.875635 20112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:30.878208 20112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6db57b5a70e24110aa83134156d3fd28: Sending DeleteTablet for 3 replicas of tablet a986dbb589b54212b390d5bdc3a92893
I20260620 14:24:30.878942 20235 tablet_service.cc:1514] Processing DeleteTablet for tablet a986dbb589b54212b390d5bdc3a92893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:30 UTC) from {username='slave'} at 127.0.0.1:47090
I20260620 14:24:30.879053 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet a986dbb589b54212b390d5bdc3a92893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:30 UTC) from {username='slave'} at 127.0.0.1:32814
I20260620 14:24:30.879106 20366 tablet_service.cc:1514] Processing DeleteTablet for tablet a986dbb589b54212b390d5bdc3a92893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:30 UTC) from {username='slave'} at 127.0.0.1:42202
I20260620 14:24:30.879433 20604 tablet_replica.cc:330] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: stopping tablet replica
I20260620 14:24:30.879433 20603 tablet_replica.cc:330] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739: stopping tablet replica
I20260620 14:24:30.879734 20603 raft_consensus.cc:2229] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:30.879734 20604 raft_consensus.cc:2229] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:30.880086 20604 raft_consensus.cc:2258] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:30.880112 20603 raft_consensus.cc:2258] T a986dbb589b54212b390d5bdc3a92893 P 0e24e9dd8d224fbdacd3d65815803739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:30.880446 20605 tablet_replica.cc:330] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: stopping tablet replica
I20260620 14:24:30.880722 20605 raft_consensus.cc:2229] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:30.880789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 20173
I20260620 14:24:30.882037 20605 raft_consensus.cc:2258] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:30.882220 20604 ts_tablet_manager.cc:1905] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:30.883709 20605 ts_tablet_manager.cc:1905] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:30.885599 20604 ts_tablet_manager.cc:1918] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:30.885700 20604 log.cc:1196] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-2/wal/wals/a986dbb589b54212b390d5bdc3a92893
I20260620 14:24:30.886006 20605 ts_tablet_manager.cc:1918] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:30.886021 20604 ts_tablet_manager.cc:1939] T a986dbb589b54212b390d5bdc3a92893 P 2fd47bb10d5b45f48570bfad2fa3dda5: Deleting consensus metadata
I20260620 14:24:30.886128 20605 log.cc:1196] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: Deleting WAL directory at /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965214882667-392-0/minicluster-data/ts-1/wal/wals/a986dbb589b54212b390d5bdc3a92893
I20260620 14:24:30.886415 20605 ts_tablet_manager.cc:1939] T a986dbb589b54212b390d5bdc3a92893 P 68b6092d49304d438c259eabd100128c: Deleting consensus metadata
I20260620 14:24:30.887396 20097 catalog_manager.cc:4841] TS 2fd47bb10d5b45f48570bfad2fa3dda5 (127.0.98.3:39933): tablet a986dbb589b54212b390d5bdc3a92893 (table test-table [id=fe18a3b8685144e6a2f8aaf9c45d4e12]) successfully deleted
I20260620 14:24:30.887611 20097 catalog_manager.cc:4841] TS 68b6092d49304d438c259eabd100128c (127.0.98.2:33851): tablet a986dbb589b54212b390d5bdc3a92893 (table test-table [id=fe18a3b8685144e6a2f8aaf9c45d4e12]) successfully deleted
W20260620 14:24:30.888700 20098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39597 (error 108)
I20260620 14:24:30.889460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 20304
I20260620 14:24:30.889788 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.889492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39597 (local address 127.0.0.1:47104)
0620 14:24:30.889623 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:30.889697 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:24:30.890547 20098 catalog_manager.cc:4584] TS 0e24e9dd8d224fbdacd3d65815803739 (127.0.98.1:39597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a986dbb589b54212b390d5bdc3a92893: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39597: connect: Connection refused (error 111)
I20260620 14:24:30.895520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 20435
I20260620 14:24:30.900846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 20082
2026-06-20T14:24:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1021 ms)
[----------] 6 tests from EncryptionPolicyTest (9992 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:30 dist-test-slave-87nc krb5kdc[20609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:30 dist-test-slave-87nc krb5kdc[20609](info): set up 2 sockets
Jun 20 14:24:30 dist-test-slave-87nc krb5kdc[20609](info): commencing operation
krb5kdc: starting...
W20260620 14:24:31.944020 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:32 dist-test-slave-87nc krb5kdc[20609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:32.066768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:32.176350 20622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:32.176642 20622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:32.176684 20622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:32.180739 20622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:32.180801 20622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:32.180820 20622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:32.180837 20622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:32.180852 20622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:32.186241 20622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:32.187350 20622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:32.188452 20622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:32.197511 20630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:32.197685 20622 server_base.cc:1034] running on GCE node
W20260620 14:24:32.197539 20628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:32.197908 20627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:32.198534 20622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:32.199674 20622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:32.200862 20622 hybrid_clock.cc:648] HybridClock initialized: now 1781965472200824 us; error 51 us; skew 500 ppm
Jun 20 14:24:32 dist-test-slave-87nc krb5kdc[20609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:32.203613 20622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:32.204645 20622 webserver.cc:460] Webserver started at http://127.0.0.1:35787/ using document root <none> and password file <none>
I20260620 14:24:32.205251 20622 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:32.205351 20622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:32.205574 20622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:32.207639 20622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "4b02698da95a4dcfb289b47a496e7228"
format_stamp: "Formatted at 2026-06-20 14:24:32 on dist-test-slave-87nc"
server_key: "e7c74f54b7ff632ffa2aa2c525ccf8d1"
server_key_iv: "bfe37870b74a72bbdc4c1157cc4cc6c6"
server_key_version: "encryptionkey@0"
I20260620 14:24:32.208307 20622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b02698da95a4dcfb289b47a496e7228"
format_stamp: "Formatted at 2026-06-20 14:24:32 on dist-test-slave-87nc"
server_key: "e7c74f54b7ff632ffa2aa2c525ccf8d1"
server_key_iv: "bfe37870b74a72bbdc4c1157cc4cc6c6"
server_key_version: "encryptionkey@0"
I20260620 14:24:32.212117 20622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:24:32.214501 20637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:32.215852 20622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:32.215996 20622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "4b02698da95a4dcfb289b47a496e7228"
format_stamp: "Formatted at 2026-06-20 14:24:32 on dist-test-slave-87nc"
server_key: "e7c74f54b7ff632ffa2aa2c525ccf8d1"
server_key_iv: "bfe37870b74a72bbdc4c1157cc4cc6c6"
server_key_version: "encryptionkey@0"
I20260620 14:24:32.216111 20622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:32.266031 20622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:32.269483 20622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:32.269726 20622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:32.277760 20622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41187
I20260620 14:24:32.277789 20689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41187 every 8 connection(s)
I20260620 14:24:32.278812 20622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:32.281646 20690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:32.283962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 20622
I20260620 14:24:32.284124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:32.284442 392 external_mini_cluster.cc:1428] Setting key cded657e9dd54905d00088ef0fe6d2fb
I20260620 14:24:32.287637 20690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228: Bootstrap starting.
I20260620 14:24:32.289849 20690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:32.290594 20690 log.cc:824] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:32.292485 20690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228: No bootstrap required, opened a new log
Jun 20 14:24:32 dist-test-slave-87nc krb5kdc[20609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:24:32.295678 20690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b02698da95a4dcfb289b47a496e7228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41187 } }
I20260620 14:24:32.295962 20690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:32.296041 20690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b02698da95a4dcfb289b47a496e7228, State: Initialized, Role: FOLLOWER
I20260620 14:24:32.296571 20690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b02698da95a4dcfb289b47a496e7228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41187 } }
I20260620 14:24:32.296753 20690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:32.296842 20690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:32.296943 20690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:32.297794 20690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b02698da95a4dcfb289b47a496e7228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41187 } }
I20260620 14:24:32.298170 20690 leader_election.cc:302] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b02698da95a4dcfb289b47a496e7228; no voters:
I20260620 14:24:32.298524 20690 leader_election.cc:288] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:32.298645 20695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:32.298902 20695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [term 1 LEADER]: Becoming Leader. State: Replica: 4b02698da95a4dcfb289b47a496e7228, State: Running, Role: LEADER
I20260620 14:24:32.299233 20695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b02698da95a4dcfb289b47a496e7228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41187 } }
I20260620 14:24:32.299885 20690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:32.300552 20695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b02698da95a4dcfb289b47a496e7228. Latest consensus state: current_term: 1 leader_uuid: "4b02698da95a4dcfb289b47a496e7228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b02698da95a4dcfb289b47a496e7228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41187 } } }
I20260620 14:24:32.300688 20695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:32.301353 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:32.285957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55433 (local address 127.0.0.1:41187)
0620 14:24:32.286427 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:32.286444 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:32.286481 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:24:32.287134 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:32.287152 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:32.287505 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:32.287901 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:32.288778 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:32.289820 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:32.290803 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:32.291294 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:32.291491 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:32.294453 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:32.294459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:32.294471 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:32.294500 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:32.297132 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:32.297727 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:32.297733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:32.297740 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:32.297871 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:32.298273 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:32.298276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:32.298277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:32.298870 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:32.299107 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:32.299950 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 14:24:32.300180 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":131,"threads_started":1}
I20260620 14:24:32.301668 20697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b02698da95a4dcfb289b47a496e7228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b02698da95a4dcfb289b47a496e7228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41187 } } }
I20260620 14:24:32.301786 20697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:32.304884 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 14:24:32.305441 20710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:32.305526 20710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:32.305598 20702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:32.306536 20702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:32.312528 20702 catalog_manager.cc:1347] Generated new cluster ID: 31faca03357e48fbb808b2947b8d734b
I20260620 14:24:32.312594 20702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:32.316969 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:32.307971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47347 (local address 127.0.0.1:41187)
0620 14:24:32.308100 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:32.308104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:32.308119 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:24:32.308153 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:32.308156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:32.308198 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:32.308283 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:32.309185 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:32.310021 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:32.311092 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:32.311492 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:32.311598 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:32.313268 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:32.313277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:32.313280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:32.313303 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:32.314924 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:32.315461 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:32.315465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:32.315467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:32.315553 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:32.316018 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:32.316022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:32.316024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:32.316202 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:32.316384 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:32.316668 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 14:24:32.316782 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:24:32.337544 20702 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:32.338510 20702 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:32.348242 20702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b02698da95a4dcfb289b47a496e7228: Generated new TSK 0
I20260620 14:24:32.349086 20702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:32.358101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 20622
W20260620 14:24:32.363379 20607 connection.cc:535] client connection to 127.0.0.1:41187 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41187): Connection reset by peer (error 104)
2026-06-20T14:24:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:32 dist-test-slave-87nc krb5kdc[20721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:32 dist-test-slave-87nc krb5kdc[20721](info): set up 2 sockets
Jun 20 14:24:32 dist-test-slave-87nc krb5kdc[20721](info): commencing operation
krb5kdc: starting...
W20260620 14:24:33.387933 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:33 dist-test-slave-87nc krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:33.510583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:33.619000 20734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:33.619274 20734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:33.619316 20734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:33.623351 20734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:33.623422 20734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:33.623441 20734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:33.623457 20734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:33.623472 20734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:33.628775 20734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:33.629781 20734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:33.630760 20734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:33.639285 20734 server_base.cc:1034] running on GCE node
W20260620 14:24:33.639237 20742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:33.639302 20740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:33.639250 20739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:33.640214 20734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:33.641319 20734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:33.642513 20734 hybrid_clock.cc:648] HybridClock initialized: now 1781965473642500 us; error 26 us; skew 500 ppm
Jun 20 14:24:33 dist-test-slave-87nc krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:33.645350 20734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:33.646266 20734 webserver.cc:460] Webserver started at http://127.0.0.1:38791/ using document root <none> and password file <none>
I20260620 14:24:33.646787 20734 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:33.646845 20734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:33.647003 20734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:33.649029 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "00f4bf2368e64bc586ac8471181877fd"
format_stamp: "Formatted at 2026-06-20 14:24:33 on dist-test-slave-87nc"
server_key: "32e766ee20dcc74386dfacdfbd5df7c4"
server_key_iv: "ee75dff114fe0fc31654529c41dabb94"
server_key_version: "encryptionkey@0"
I20260620 14:24:33.649560 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "00f4bf2368e64bc586ac8471181877fd"
format_stamp: "Formatted at 2026-06-20 14:24:33 on dist-test-slave-87nc"
server_key: "32e766ee20dcc74386dfacdfbd5df7c4"
server_key_iv: "ee75dff114fe0fc31654529c41dabb94"
server_key_version: "encryptionkey@0"
I20260620 14:24:33.653079 20734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:33.655449 20749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:33.656862 20734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:33.657035 20734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "00f4bf2368e64bc586ac8471181877fd"
format_stamp: "Formatted at 2026-06-20 14:24:33 on dist-test-slave-87nc"
server_key: "32e766ee20dcc74386dfacdfbd5df7c4"
server_key_iv: "ee75dff114fe0fc31654529c41dabb94"
server_key_version: "encryptionkey@0"
I20260620 14:24:33.657183 20734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:33.679342 20734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:33.682788 20734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:33.683007 20734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:33.690167 20734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37725
I20260620 14:24:33.690192 20801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37725 every 8 connection(s)
I20260620 14:24:33.691104 20734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:33.693917 20802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:33.696666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 20734
I20260620 14:24:33.696774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:33.697079 392 external_mini_cluster.cc:1428] Setting key 18cd4cc40af6ed69acf586f59777ddee
I20260620 14:24:33.699946 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd: Bootstrap starting.
I20260620 14:24:33.702591 20802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:33.703394 20802 log.cc:824] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:33.705384 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd: No bootstrap required, opened a new log
I20260620 14:24:33.706796 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:33.698459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56021 (local address 127.0.0.1:37725)
0620 14:24:33.698885 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:33.698907 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:33.698950 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:24:33.699715 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:33.699737 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:33.700199 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:33.700546 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:33.700951 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.702410 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.703326 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.704017 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.704211 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:33.705114 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:33.705126 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:33.705139 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:33.705556 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:33.705619 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:33.705643 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:33.705786 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:33.705974 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:24:33.706115 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
I20260620 14:24:33.709697 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:33.710072 20802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f4bf2368e64bc586ac8471181877fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } }
I20260620 14:24:33.710314 20802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:33.710391 20802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f4bf2368e64bc586ac8471181877fd, State: Initialized, Role: FOLLOWER
I20260620 14:24:33.710800 20802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f4bf2368e64bc586ac8471181877fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } }
I20260620 14:24:33.710951 20802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:33.711040 20802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:33.711148 20802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:33.712306 20802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f4bf2368e64bc586ac8471181877fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } }
I20260620 14:24:33.712651 20802 leader_election.cc:302] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00f4bf2368e64bc586ac8471181877fd; no voters:
I20260620 14:24:33.712980 20802 leader_election.cc:288] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:33.713044 20812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:33.713232 20812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [term 1 LEADER]: Becoming Leader. State: Replica: 00f4bf2368e64bc586ac8471181877fd, State: Running, Role: LEADER
I20260620 14:24:33.713567 20812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f4bf2368e64bc586ac8471181877fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } }
I20260620 14:24:33.714231 20802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:33.714687 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00f4bf2368e64bc586ac8471181877fd. Latest consensus state: current_term: 1 leader_uuid: "00f4bf2368e64bc586ac8471181877fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f4bf2368e64bc586ac8471181877fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } } }
I20260620 14:24:33.714847 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:33.715276 20813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00f4bf2368e64bc586ac8471181877fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f4bf2368e64bc586ac8471181877fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } } }
I20260620 14:24:33.715427 20813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:33.715667 20821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:33.716431 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:33.711865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48069 (local address 127.0.0.1:37725)
0620 14:24:33.711996 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:33.712000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:33.712116 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:24:33.712219 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:33.712222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:33.712272 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:33.712374 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:33.712631 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.713359 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.714279 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.714608 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.714717 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:33.715634 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:33.715637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:33.715639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:33.715650 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:33.715672 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:33.715682 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:33.715803 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:33.716257 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 14:24:33.716294 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:24:33.718701 20821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:33.724308 20821 catalog_manager.cc:1347] Generated new cluster ID: 182aed1e75854e4287f80c48c55cb02f
I20260620 14:24:33.724398 20821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:33.732981 20821 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:33.734208 20821 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:33.747794 20821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00f4bf2368e64bc586ac8471181877fd: Generated new TSK 0
I20260620 14:24:33.748525 20821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:33.758687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 20734
W20260620 14:24:33.763965 20719 connection.cc:535] client connection to 127.0.0.1:37725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37725): Connection reset by peer (error 104)
2026-06-20T14:24:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:33 dist-test-slave-87nc krb5kdc[20833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:33 dist-test-slave-87nc krb5kdc[20833](info): set up 2 sockets
Jun 20 14:24:33 dist-test-slave-87nc krb5kdc[20833](info): commencing operation
krb5kdc: starting...
W20260620 14:24:34.800016 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:34 dist-test-slave-87nc krb5kdc[20833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:34.922976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:35.034036 20846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:35.034358 20846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:35.034436 20846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:35.038465 20846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:35.038587 20846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:35.038627 20846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:35.038663 20846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:35.038691 20846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:35.044063 20846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:35.045195 20846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:35.046239 20846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:35.054811 20852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:35.054833 20851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:35.054833 20854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:35.054872 20846 server_base.cc:1034] running on GCE node
I20260620 14:24:35.055562 20846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:35.056638 20846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:35.057844 20846 hybrid_clock.cc:648] HybridClock initialized: now 1781965475057826 us; error 37 us; skew 500 ppm
Jun 20 14:24:35 dist-test-slave-87nc krb5kdc[20833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:35.060559 20846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:35.061491 20846 webserver.cc:460] Webserver started at http://127.0.0.1:37329/ using document root <none> and password file <none>
I20260620 14:24:35.062047 20846 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:35.062107 20846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:35.062325 20846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:35.064505 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "a6b840b48f6441c3bf2fa322a5331324"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-87nc"
server_key: "8b992c5e31aa2ccefef15b644fafa3e2"
server_key_iv: "e491bfa54ab99b4a70303e8e171fd430"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.065083 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6b840b48f6441c3bf2fa322a5331324"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-87nc"
server_key: "8b992c5e31aa2ccefef15b644fafa3e2"
server_key_iv: "e491bfa54ab99b4a70303e8e171fd430"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.068884 20846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:24:35.071338 20861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:35.072726 20846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:24:35.072849 20846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "a6b840b48f6441c3bf2fa322a5331324"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-87nc"
server_key: "8b992c5e31aa2ccefef15b644fafa3e2"
server_key_iv: "e491bfa54ab99b4a70303e8e171fd430"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.073014 20846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:35.080070 20846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:35.082976 20846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:35.083132 20846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:35.090099 20846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41073
I20260620 14:24:35.090121 20913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41073 every 8 connection(s)
I20260620 14:24:35.091042 20846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:35.093894 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:35.098953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 20846
I20260620 14:24:35.099125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:35.099400 392 external_mini_cluster.cc:1428] Setting key a1b306741b8006e4d4db714e658589c8
I20260620 14:24:35.099418 20914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324: Bootstrap starting.
I20260620 14:24:35.102094 20914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:35.102949 20914 log.cc:824] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:35.105051 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:35.100728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55139 (local address 127.0.0.1:41073)
0620 14:24:35.101141 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:35.101158 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:35.101200 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:24:35.102004 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:35.102031 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:35.102412 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:35.102774 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:35.103545 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:35.103549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:35.103561 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:35.103570 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:35.103620 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:35.103638 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:35.103816 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:35.104028 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:24:35.104172 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260620 14:24:35.105182 20914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324: No bootstrap required, opened a new log
I20260620 14:24:35.107755 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:35.109236 20914 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b840b48f6441c3bf2fa322a5331324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260620 14:24:35.109503 20914 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:35.109602 20914 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b840b48f6441c3bf2fa322a5331324, State: Initialized, Role: FOLLOWER
I20260620 14:24:35.110076 20914 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b840b48f6441c3bf2fa322a5331324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260620 14:24:35.110234 20914 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:35.110302 20914 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:35.110411 20914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:35.111462 20914 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b840b48f6441c3bf2fa322a5331324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260620 14:24:35.111646 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:35.109745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51133 (local address 127.0.0.1:41073)
0620 14:24:35.109845 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:35.109850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:35.110122 (+ 272us) server_negotiation.cc:407] Connection header received
0620 14:24:35.110281 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:35.110284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:35.110326 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:35.110412 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:35.111181 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:35.111184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:35.111187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:35.111198 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:35.111227 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:35.111238 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:35.111339 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:35.111373 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:24:35.111415 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:24:35.111904 20914 leader_election.cc:302] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6b840b48f6441c3bf2fa322a5331324; no voters:
I20260620 14:24:35.112255 20914 leader_election.cc:288] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:35.112324 20924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:35.112526 20924 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [term 1 LEADER]: Becoming Leader. State: Replica: a6b840b48f6441c3bf2fa322a5331324, State: Running, Role: LEADER
I20260620 14:24:35.112826 20924 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b840b48f6441c3bf2fa322a5331324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260620 14:24:35.113598 20914 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:35.114048 20924 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6b840b48f6441c3bf2fa322a5331324. Latest consensus state: current_term: 1 leader_uuid: "a6b840b48f6441c3bf2fa322a5331324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b840b48f6441c3bf2fa322a5331324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } } }
I20260620 14:24:35.114218 20924 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:35.114442 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6b840b48f6441c3bf2fa322a5331324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b840b48f6441c3bf2fa322a5331324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } } }
I20260620 14:24:35.114552 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:35.114642 20929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:35.118055 20929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:35.123884 20929 catalog_manager.cc:1347] Generated new cluster ID: d659e0167e3d4da595b26992a0247fc2
I20260620 14:24:35.123966 20929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:35.166239 20929 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:35.166599 20929 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:35.175114 20929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6b840b48f6441c3bf2fa322a5331324: Generated new TSK 0
I20260620 14:24:35.175854 20929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:35.189833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 20846
2026-06-20T14:24:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:35 dist-test-slave-87nc krb5kdc[20945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:35 dist-test-slave-87nc krb5kdc[20945](info): set up 2 sockets
Jun 20 14:24:35 dist-test-slave-87nc krb5kdc[20945](info): commencing operation
krb5kdc: starting...
W20260620 14:24:36.275947 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:36 dist-test-slave-87nc krb5kdc[20945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:36.397010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:36.508180 20958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:36.508518 20958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:36.508606 20958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:36.512887 20958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:36.512985 20958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:36.513024 20958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:36.513084 20958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:36.513126 20958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:36.518626 20958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:36.519878 20958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:36.520963 20958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:36.529706 20966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:36.529778 20963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:36.529883 20958 server_base.cc:1034] running on GCE node
W20260620 14:24:36.529709 20964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:36.530544 20958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:36.531672 20958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:36.532840 20958 hybrid_clock.cc:648] HybridClock initialized: now 1781965476532808 us; error 42 us; skew 500 ppm
Jun 20 14:24:36 dist-test-slave-87nc krb5kdc[20945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:36.535584 20958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:36.536576 20958 webserver.cc:460] Webserver started at http://127.0.0.1:34757/ using document root <none> and password file <none>
I20260620 14:24:36.537142 20958 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:36.537204 20958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:36.537418 20958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:36.539498 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "6ba9043ab1ba4207a4c86451c3af41bb"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-87nc"
server_key: "4c359aa9e19bf369bce0290a0885209c"
server_key_iv: "cf9330c048d9b7cab6fab1e1bae782c2"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.540117 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ba9043ab1ba4207a4c86451c3af41bb"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-87nc"
server_key: "4c359aa9e19bf369bce0290a0885209c"
server_key_iv: "cf9330c048d9b7cab6fab1e1bae782c2"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.544008 20958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 14:24:36.546433 20973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.547727 20958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:36.547909 20958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "6ba9043ab1ba4207a4c86451c3af41bb"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-87nc"
server_key: "4c359aa9e19bf369bce0290a0885209c"
server_key_iv: "cf9330c048d9b7cab6fab1e1bae782c2"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.548038 20958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:36.570990 20958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:36.574162 20958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:36.574378 20958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:36.581570 20958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35971
I20260620 14:24:36.581585 21025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35971 every 8 connection(s)
I20260620 14:24:36.582620 20958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:36.583496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 20958
I20260620 14:24:36.583659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:36.583968 392 external_mini_cluster.cc:1428] Setting key 661fb083cbb1d94396ca032022af0ab6
I20260620 14:24:36.585772 21026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:36.592412 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb: Bootstrap starting.
Jun 20 14:24:36 dist-test-slave-87nc krb5kdc[20945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:24:36.595077 21026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:36.596009 21026 log.cc:824] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:36.598232 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb: No bootstrap required, opened a new log
I20260620 14:24:36.598642 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.585294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54615 (local address 127.0.0.1:35971)
0620 14:24:36.585695 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.585713 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.585751 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:24:36.586421 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.586437 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.586790 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.587077 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:36.588030 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.589032 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.590209 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.590691 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.590817 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.593622 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:36.593629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:36.593648 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:36.593684 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:36.595601 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:36.596059 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:36.596065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:36.596070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:36.596213 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:36.596511 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:36.596514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:36.596515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:36.597004 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:36.597191 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.597417 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.597704 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":142,"threads_started":1}
I20260620 14:24:36.601868 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:36.601893 21026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba9043ab1ba4207a4c86451c3af41bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260620 14:24:36.602191 21026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:36.602258 21026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba9043ab1ba4207a4c86451c3af41bb, State: Initialized, Role: FOLLOWER
I20260620 14:24:36.602703 21026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba9043ab1ba4207a4c86451c3af41bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260620 14:24:36.602856 21026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:36.602923 21026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:36.603050 21026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:36.604269 21026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba9043ab1ba4207a4c86451c3af41bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260620 14:24:36.604650 21026 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba9043ab1ba4207a4c86451c3af41bb; no voters:
I20260620 14:24:36.604998 21026 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:36.605080 21036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:36.605257 21036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [term 1 LEADER]: Becoming Leader. State: Replica: 6ba9043ab1ba4207a4c86451c3af41bb, State: Running, Role: LEADER
I20260620 14:24:36.605602 21036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba9043ab1ba4207a4c86451c3af41bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260620 14:24:36.606309 21026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:36.607308 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ba9043ab1ba4207a4c86451c3af41bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba9043ab1ba4207a4c86451c3af41bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } } }
I20260620 14:24:36.607432 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:36.607822 21038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba9043ab1ba4207a4c86451c3af41bb. Latest consensus state: current_term: 1 leader_uuid: "6ba9043ab1ba4207a4c86451c3af41bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba9043ab1ba4207a4c86451c3af41bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } } }
I20260620 14:24:36.607991 21038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:36.608758 21045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:36.612407 21045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:36.612920 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.603945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41585 (local address 127.0.0.1:35971)
0620 14:24:36.604094 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.604100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.604236 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:24:36.604317 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.604320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.604372 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.604454 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:36.605232 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.605885 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.606893 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.607223 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.607326 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.609199 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:36.609203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:36.609205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:36.609229 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:36.610791 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:36.611264 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:36.611268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:36.611270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:36.611350 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:36.611663 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:36.611666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:36.611667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:36.611899 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:36.612017 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.612646 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.612767 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:24:36.619287 21045 catalog_manager.cc:1347] Generated new cluster ID: 7e020ad373064666a321c5b9b056af03
I20260620 14:24:36.619369 21045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:36.640774 21045 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:36.641963 21045 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:36.654300 21045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ba9043ab1ba4207a4c86451c3af41bb: Generated new TSK 0
I20260620 14:24:36.655061 21045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:36.664877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 20958
2026-06-20T14:24:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:36 dist-test-slave-87nc krb5kdc[21057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:36 dist-test-slave-87nc krb5kdc[21057](info): set up 2 sockets
Jun 20 14:24:36 dist-test-slave-87nc krb5kdc[21057](info): commencing operation
krb5kdc: starting...
W20260620 14:24:37.699996 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:37 dist-test-slave-87nc krb5kdc[21057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:37.825112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:37.936518 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:37.936857 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:37.936940 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:37.941130 21070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:37.941237 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:37.941275 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:37.941340 21070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:37.941380 21070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:37.946871 21070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:37.948113 21070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:37.949188 21070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:37.957898 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:37.958002 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:37.958002 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:37.958171 21070 server_base.cc:1034] running on GCE node
I20260620 14:24:37.958760 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:37.959880 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:37.961074 21070 hybrid_clock.cc:648] HybridClock initialized: now 1781965477961047 us; error 45 us; skew 500 ppm
Jun 20 14:24:37 dist-test-slave-87nc krb5kdc[21057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:37.966941 21070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:37.968044 21070 webserver.cc:460] Webserver started at http://127.0.0.1:43983/ using document root <none> and password file <none>
I20260620 14:24:37.968627 21070 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:37.968726 21070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:37.968961 21070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:37.971079 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "a03a926b4e6843c6893f772da031bddd"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-87nc"
server_key: "b0686761760a2e7b3521318d659007b3"
server_key_iv: "9bde881edbd200265bb607bbf6e6db0e"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.971650 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "a03a926b4e6843c6893f772da031bddd"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-87nc"
server_key: "b0686761760a2e7b3521318d659007b3"
server_key_iv: "9bde881edbd200265bb607bbf6e6db0e"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.975400 21070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:24:37.977867 21085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:37.979059 21070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:37.979216 21070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "a03a926b4e6843c6893f772da031bddd"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-87nc"
server_key: "b0686761760a2e7b3521318d659007b3"
server_key_iv: "9bde881edbd200265bb607bbf6e6db0e"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.979339 21070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:38.012344 21070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:38.015195 21070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:38.015415 21070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:38.022713 21070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45097
I20260620 14:24:38.022732 21137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45097 every 8 connection(s)
I20260620 14:24:38.023654 21070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:38.026328 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:38.031566 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd: Bootstrap starting.
I20260620 14:24:38.031980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 21070
I20260620 14:24:38.032158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:38.032423 392 external_mini_cluster.cc:1428] Setting key 9a424d4b5c2004511f0b1ba74fba2d99
I20260620 14:24:38.034291 21138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:38.035023 21138 log.cc:824] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:38.037053 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd: No bootstrap required, opened a new log
I20260620 14:24:38.041410 21138 raft_consensus.cc:348] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a926b4e6843c6893f772da031bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45097 } }
I20260620 14:24:38.041654 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:38.033786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33477 (local address 127.0.0.1:45097)
0620 14:24:38.034224 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:38.034239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:38.034272 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:24:38.034954 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:38.034971 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:38.035397 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:38.035812 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:38.036143 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:38.037134 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:38.038374 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:38.038846 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:38.038963 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:38.039917 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:38.039922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:38.039935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:38.040340 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:38.040393 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:38.040416 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:38.040491 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:38.040663 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:24:38.040897 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":134,"threads_started":1}
I20260620 14:24:38.041668 21138 raft_consensus.cc:374] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:38.041844 21138 raft_consensus.cc:729] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03a926b4e6843c6893f772da031bddd, State: Initialized, Role: FOLLOWER
I20260620 14:24:38.042341 21138 consensus_queue.cc:260] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a926b4e6843c6893f772da031bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45097 } }
I20260620 14:24:38.042500 21138 raft_consensus.cc:388] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:38.042563 21138 raft_consensus.cc:482] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:38.042647 21138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:38.043606 21138 raft_consensus.cc:504] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a926b4e6843c6893f772da031bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45097 } }
I20260620 14:24:38.044023 21138 leader_election.cc:302] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a03a926b4e6843c6893f772da031bddd; no voters:
I20260620 14:24:38.044366 21138 leader_election.cc:288] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:38.044498 21143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:38.044740 21143 raft_consensus.cc:686] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [term 1 LEADER]: Becoming Leader. State: Replica: a03a926b4e6843c6893f772da031bddd, State: Running, Role: LEADER
I20260620 14:24:38.044857 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:38.045138 21143 consensus_queue.cc:237] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a926b4e6843c6893f772da031bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45097 } }
I20260620 14:24:38.045742 21138 sys_catalog.cc:564] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:38.046715 21144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a03a926b4e6843c6893f772da031bddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a926b4e6843c6893f772da031bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45097 } } }
I20260620 14:24:38.046859 21144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:38.047202 21145 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a03a926b4e6843c6893f772da031bddd. Latest consensus state: current_term: 1 leader_uuid: "a03a926b4e6843c6893f772da031bddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a926b4e6843c6893f772da031bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45097 } } }
I20260620 14:24:38.047319 21145 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:38.047276 21156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:38.051085 21156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:38.052179 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:38.047464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41267 (local address 127.0.0.1:45097)
0620 14:24:38.047589 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:38.047597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:38.047615 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:24:38.047855 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:38.047859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:38.047909 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:38.047990 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:38.048523 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:38.049258 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:38.050378 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:38.050714 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:38.050837 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:38.051701 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:38.051705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:38.051708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:38.051720 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:38.051747 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:38.051760 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:38.051865 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:38.051989 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:24:38.052038 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:24:38.056654 21156 catalog_manager.cc:1347] Generated new cluster ID: e43eea457eec47658daefd03c6ffd6ac
I20260620 14:24:38.056715 21156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:38.066152 21156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:38.067041 21156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:38.071024 21156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a03a926b4e6843c6893f772da031bddd: Generated new TSK 0
I20260620 14:24:38.071635 21156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:38.080009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 21070
2026-06-20T14:24:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:38 dist-test-slave-87nc krb5kdc[21169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:38 dist-test-slave-87nc krb5kdc[21169](info): set up 2 sockets
Jun 20 14:24:38 dist-test-slave-87nc krb5kdc[21169](info): commencing operation
krb5kdc: starting...
W20260620 14:24:39.120229 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:39 dist-test-slave-87nc krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:39.239547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:39.349327 21182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:39.349601 21182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:39.349642 21182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:39.353693 21182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:39.353755 21182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:39.353775 21182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:39.353791 21182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:39.353806 21182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:39.359320 21182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:39.360404 21182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:39.361434 21182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:39.370106 21188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:39.370080 21190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:39.370080 21187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:39.370151 21182 server_base.cc:1034] running on GCE node
I20260620 14:24:39.371007 21182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:39.372165 21182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:39.373358 21182 hybrid_clock.cc:648] HybridClock initialized: now 1781965479373347 us; error 31 us; skew 500 ppm
Jun 20 14:24:39 dist-test-slave-87nc krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:39.376148 21182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:39.377110 21182 webserver.cc:460] Webserver started at http://127.0.0.1:46023/ using document root <none> and password file <none>
I20260620 14:24:39.377669 21182 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:39.377771 21182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:39.377998 21182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:39.380107 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "2f9717140fd04c8aa87339bd766ad29a"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-87nc"
server_key: "dc4ff71b506a49f676321e86e288e1f1"
server_key_iv: "33075b9fd3ba96e0fcd275809de91930"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.380668 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f9717140fd04c8aa87339bd766ad29a"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-87nc"
server_key: "dc4ff71b506a49f676321e86e288e1f1"
server_key_iv: "33075b9fd3ba96e0fcd275809de91930"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.384446 21182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:24:39.386830 21197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:39.388168 21182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 14:24:39.388299 21182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "2f9717140fd04c8aa87339bd766ad29a"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-87nc"
server_key: "dc4ff71b506a49f676321e86e288e1f1"
server_key_iv: "33075b9fd3ba96e0fcd275809de91930"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.388413 21182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:39.392987 21182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:39.396297 21182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:39.396492 21182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:39.403741 21182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45849
I20260620 14:24:39.403790 21249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45849 every 8 connection(s)
I20260620 14:24:39.404724 21182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:39.405653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 21182
I20260620 14:24:39.405815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:39.406111 392 external_mini_cluster.cc:1428] Setting key f665dd317a4063dc5c1834acc8a2cbdb
I20260620 14:24:39.408088 21250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:39.411381 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:39.407501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48671 (local address 127.0.0.1:45849)
0620 14:24:39.407989 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:39.408004 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:39.408040 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:24:39.408716 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:39.408732 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:39.409133 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:39.409466 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:39.410253 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:39.410258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:39.410270 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:39.410280 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:39.410330 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:39.410348 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:39.410486 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:39.410555 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:24:39.410703 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":171,"threads_started":1}
I20260620 14:24:39.414033 21250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a: Bootstrap starting.
I20260620 14:24:39.414088 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:39.416401 21250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:39.417265 21250 log.cc:824] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:39.417661 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:39.415971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60457 (local address 127.0.0.1:45849)
0620 14:24:39.416096 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:39.416100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:39.416325 (+ 225us) server_negotiation.cc:407] Connection header received
0620 14:24:39.416450 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:39.416452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:39.416505 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:39.416600 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:39.417292 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:39.417295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:39.417298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:39.417309 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:39.417335 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:39.417348 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:39.417420 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:39.417501 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:24:39.417538 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:24:39.419112 21250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a: No bootstrap required, opened a new log
I20260620 14:24:39.422446 21250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9717140fd04c8aa87339bd766ad29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260620 14:24:39.422667 21250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:39.422703 21250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f9717140fd04c8aa87339bd766ad29a, State: Initialized, Role: FOLLOWER
I20260620 14:24:39.423048 21250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9717140fd04c8aa87339bd766ad29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260620 14:24:39.423153 21250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:39.423206 21250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:39.423272 21250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:39.424190 21250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9717140fd04c8aa87339bd766ad29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260620 14:24:39.424475 21250 leader_election.cc:302] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f9717140fd04c8aa87339bd766ad29a; no voters:
I20260620 14:24:39.424717 21250 leader_election.cc:288] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:39.424839 21260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:39.425093 21260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [term 1 LEADER]: Becoming Leader. State: Replica: 2f9717140fd04c8aa87339bd766ad29a, State: Running, Role: LEADER
I20260620 14:24:39.425441 21260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9717140fd04c8aa87339bd766ad29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260620 14:24:39.425997 21250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:39.427139 21261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f9717140fd04c8aa87339bd766ad29a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9717140fd04c8aa87339bd766ad29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260620 14:24:39.427276 21261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:39.427572 21262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f9717140fd04c8aa87339bd766ad29a. Latest consensus state: current_term: 1 leader_uuid: "2f9717140fd04c8aa87339bd766ad29a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9717140fd04c8aa87339bd766ad29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260620 14:24:39.427644 21269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:39.428231 21262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:39.431298 21269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:39.436580 21269 catalog_manager.cc:1347] Generated new cluster ID: 450804991bd64ef597c7bb868cbf6738
I20260620 14:24:39.436643 21269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:39.463619 21269 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:39.463989 21269 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:39.472988 21269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f9717140fd04c8aa87339bd766ad29a: Generated new TSK 0
I20260620 14:24:39.473826 21269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:39.488148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 21182
2026-06-20T14:24:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:39 dist-test-slave-87nc krb5kdc[21281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:39 dist-test-slave-87nc krb5kdc[21281](info): set up 2 sockets
Jun 20 14:24:39 dist-test-slave-87nc krb5kdc[21281](info): commencing operation
krb5kdc: starting...
W20260620 14:24:40.528028 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:40 dist-test-slave-87nc krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:40.650799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:40.765581 21294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:40.765892 21294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:40.765939 21294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:40.770429 21294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:40.770524 21294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:40.770545 21294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:40.770563 21294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:40.770578 21294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:40.776427 21294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:40.777563 21294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:40.778558 21294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:40.786818 21300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:40.786829 21302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:40.786819 21299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:40.787191 21294 server_base.cc:1034] running on GCE node
I20260620 14:24:40.787750 21294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:40.788906 21294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:40.790086 21294 hybrid_clock.cc:648] HybridClock initialized: now 1781965480790047 us; error 52 us; skew 500 ppm
Jun 20 14:24:40 dist-test-slave-87nc krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:40.793164 21294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:40.794108 21294 webserver.cc:460] Webserver started at http://127.0.0.1:42181/ using document root <none> and password file <none>
I20260620 14:24:40.794658 21294 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:40.794718 21294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:40.794883 21294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:40.796952 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "8760d4a4fdd94ead94ca37720722bdb4"
format_stamp: "Formatted at 2026-06-20 14:24:40 on dist-test-slave-87nc"
server_key: "e6c462e478750affa6861dab9afc5022"
server_key_iv: "132bada69070bd0621c907a83530729b"
server_key_version: "encryptionkey@0"
I20260620 14:24:40.797588 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "8760d4a4fdd94ead94ca37720722bdb4"
format_stamp: "Formatted at 2026-06-20 14:24:40 on dist-test-slave-87nc"
server_key: "e6c462e478750affa6861dab9afc5022"
server_key_iv: "132bada69070bd0621c907a83530729b"
server_key_version: "encryptionkey@0"
I20260620 14:24:40.801308 21294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:24:40.803570 21309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:40.804843 21294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:40.804986 21294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "8760d4a4fdd94ead94ca37720722bdb4"
format_stamp: "Formatted at 2026-06-20 14:24:40 on dist-test-slave-87nc"
server_key: "e6c462e478750affa6861dab9afc5022"
server_key_iv: "132bada69070bd0621c907a83530729b"
server_key_version: "encryptionkey@0"
I20260620 14:24:40.805101 21294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:40.816718 21294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:40.819571 21294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:40.819820 21294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:40.826923 21294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45127
I20260620 14:24:40.826983 21361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45127 every 8 connection(s)
I20260620 14:24:40.827893 21294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:40.830606 21362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:40.835913 21362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4: Bootstrap starting.
I20260620 14:24:40.837193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 21294
I20260620 14:24:40.837334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:40.837613 392 external_mini_cluster.cc:1428] Setting key ccee48ce525f20d58cac3781b0d67a08
I20260620 14:24:40.838716 21362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:40.839807 21362 log.cc:824] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:40.841984 21362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4: No bootstrap required, opened a new log
I20260620 14:24:40.845999 21362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760d4a4fdd94ead94ca37720722bdb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260620 14:24:40.846282 21362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:40.846372 21362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8760d4a4fdd94ead94ca37720722bdb4, State: Initialized, Role: FOLLOWER
Jun 20 14:24:40 dist-test-slave-87nc krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:24:40.846957 21362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760d4a4fdd94ead94ca37720722bdb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260620 14:24:40.847323 21362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:40.847492 21362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:40.847631 21362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:40.848939 21362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760d4a4fdd94ead94ca37720722bdb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260620 14:24:40.849308 21362 leader_election.cc:302] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8760d4a4fdd94ead94ca37720722bdb4; no voters:
I20260620 14:24:40.849632 21362 leader_election.cc:288] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:40.849781 21367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:40.850024 21367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [term 1 LEADER]: Becoming Leader. State: Replica: 8760d4a4fdd94ead94ca37720722bdb4, State: Running, Role: LEADER
I20260620 14:24:40.850445 21367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760d4a4fdd94ead94ca37720722bdb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260620 14:24:40.851080 21362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:40.852273 21368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8760d4a4fdd94ead94ca37720722bdb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760d4a4fdd94ead94ca37720722bdb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } } }
I20260620 14:24:40.852474 21368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:40.852941 21376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:40.853073 21369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8760d4a4fdd94ead94ca37720722bdb4. Latest consensus state: current_term: 1 leader_uuid: "8760d4a4fdd94ead94ca37720722bdb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760d4a4fdd94ead94ca37720722bdb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } } }
I20260620 14:24:40.854063 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:40.838750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:37183 (local address 127.0.0.1:45127)
0620 14:24:40.839383 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:40.839397 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:40.839432 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:24:40.840170 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:40.840188 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:40.840535 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:40.840872 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:40.841867 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.842902 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.843943 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.844469 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.844646 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:40.847426 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:40.847432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:40.847445 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:40.847473 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:40.849597 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.850310 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.850316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.850322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.850460 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.850808 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.850812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.850813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.851331 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:40.851579 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:40.852432 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 14:24:40.852663 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":219,"threads_started":1}
I20260620 14:24:40.853297 21369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:40.856501 21376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:40.859606 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:40.862905 21376 catalog_manager.cc:1347] Generated new cluster ID: f29d8de8610640f190a0d30c0bd0a22c
I20260620 14:24:40.862983 21376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:40.872942 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:40.861749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:41977 (local address 127.0.0.1:45127)
0620 14:24:40.861861 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:40.861865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:40.864128 (+ 2263us) server_negotiation.cc:407] Connection header received
0620 14:24:40.864255 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:40.864257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:40.864306 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:40.864383 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:40.865226 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.866013 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.867185 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.867562 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.867654 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:40.869401 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:40.869405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:40.869408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:40.869439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:40.871082 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.871668 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.871671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.871673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.871753 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.872085 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.872089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.872091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.872293 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:40.872418 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:40.872686 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:24:40.872779 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":33}
I20260620 14:24:40.877877 21376 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:40.878908 21376 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:40.885324 21376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8760d4a4fdd94ead94ca37720722bdb4: Generated new TSK 0
I20260620 14:24:40.886025 21376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:40.894809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 21294
W20260620 14:24:40.900200 21279 connection.cc:535] client connection to 127.0.0.1:45127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45127): Connection reset by peer (error 104)
2026-06-20T14:24:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:40 dist-test-slave-87nc krb5kdc[21393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:40 dist-test-slave-87nc krb5kdc[21393](info): set up 2 sockets
Jun 20 14:24:40 dist-test-slave-87nc krb5kdc[21393](info): commencing operation
krb5kdc: starting...
W20260620 14:24:41.924115 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:42 dist-test-slave-87nc krb5kdc[21393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:42.040879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:42.151096 21406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:42.151428 21406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:42.151502 21406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:42.155597 21406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:42.155714 21406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:42.155753 21406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:42.155812 21406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:42.155875 21406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:42.161361 21406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:42.162470 21406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:42.163520 21406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:42.172381 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.172590 21406 server_base.cc:1034] running on GCE node
W20260620 14:24:42.172382 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.172380 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.173514 21406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:42.174777 21406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:42.175980 21406 hybrid_clock.cc:648] HybridClock initialized: now 1781965482175960 us; error 36 us; skew 500 ppm
Jun 20 14:24:42 dist-test-slave-87nc krb5kdc[21393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:42.179051 21406 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:42.180122 21406 webserver.cc:460] Webserver started at http://127.0.0.1:42657/ using document root <none> and password file <none>
I20260620 14:24:42.180711 21406 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.180805 21406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.181048 21406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:42.183210 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "c8e33affc0894643a92139ac6623b7a8"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-87nc"
server_key: "dada7ec5e970feb344071b91c488febc"
server_key_iv: "cd918f06ab26380a649ae917860dc780"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.183964 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8e33affc0894643a92139ac6623b7a8"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-87nc"
server_key: "dada7ec5e970feb344071b91c488febc"
server_key_iv: "cd918f06ab26380a649ae917860dc780"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.187906 21406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:24:42.190459 21421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.191723 21406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:42.191926 21406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "c8e33affc0894643a92139ac6623b7a8"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-87nc"
server_key: "dada7ec5e970feb344071b91c488febc"
server_key_iv: "cd918f06ab26380a649ae917860dc780"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.192060 21406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.203938 21406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:42.207470 21406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:42.207692 21406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:42.214975 21406 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44749
I20260620 14:24:42.214999 21473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44749 every 8 connection(s)
I20260620 14:24:42.216038 21406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:42.217486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 21406
I20260620 14:24:42.217592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:42.217875 392 external_mini_cluster.cc:1428] Setting key f0f054efc35ad4996e2d31bbeea2d496
I20260620 14:24:42.219085 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:42.224782 21474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8: Bootstrap starting.
I20260620 14:24:42.226950 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.219297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:37629 (local address 127.0.0.1:44749)
0620 14:24:42.219745 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.219760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.219866 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:24:42.220506 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.220521 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.220872 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.221212 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.221621 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.222605 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.223508 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.224061 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.224235 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.225241 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.225246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.225260 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:42.225657 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.225711 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:42.225738 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.225894 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.226074 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.226215 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":157,"threads_started":1}
I20260620 14:24:42.227183 21474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:42.228017 21474 log.cc:824] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:42.230113 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:42.230356 21474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8: No bootstrap required, opened a new log
I20260620 14:24:42.234843 21474 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e33affc0894643a92139ac6623b7a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } }
I20260620 14:24:42.235150 21474 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:42.235219 21474 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e33affc0894643a92139ac6623b7a8, State: Initialized, Role: FOLLOWER
I20260620 14:24:42.235746 21474 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e33affc0894643a92139ac6623b7a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } }
I20260620 14:24:42.235961 21474 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:42.236053 21474 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:42.236183 21474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:42.236565 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.232205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:55605 (local address 127.0.0.1:44749)
0620 14:24:42.232321 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.232325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.232590 (+ 265us) server_negotiation.cc:407] Connection header received
0620 14:24:42.232725 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.232727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.232764 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.232829 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.233134 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.233641 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.234741 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.234966 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.235047 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.235990 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.235996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.235999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:42.236014 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.236045 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:42.236058 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.236174 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.236362 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.236408 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:24:42.237362 21474 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e33affc0894643a92139ac6623b7a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } }
I20260620 14:24:42.237712 21474 leader_election.cc:302] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8e33affc0894643a92139ac6623b7a8; no voters:
I20260620 14:24:42.238034 21474 leader_election.cc:288] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:42.238171 21484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:42.238420 21484 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [term 1 LEADER]: Becoming Leader. State: Replica: c8e33affc0894643a92139ac6623b7a8, State: Running, Role: LEADER
I20260620 14:24:42.238790 21484 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e33affc0894643a92139ac6623b7a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } }
I20260620 14:24:42.239331 21474 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:42.240742 21486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e33affc0894643a92139ac6623b7a8. Latest consensus state: current_term: 1 leader_uuid: "c8e33affc0894643a92139ac6623b7a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e33affc0894643a92139ac6623b7a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } } }
I20260620 14:24:42.240873 21486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:42.241052 21485 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8e33affc0894643a92139ac6623b7a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e33affc0894643a92139ac6623b7a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } } }
I20260620 14:24:42.241230 21485 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8 [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:42.244242 21499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:42.244344 21499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:42.244426 21498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:42.245113 21498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:42.250658 21498 catalog_manager.cc:1347] Generated new cluster ID: e9e6d58af4534f999ad22db4721c0a64
I20260620 14:24:42.250734 21498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:42.260236 21498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:42.261457 21498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:42.266085 21498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8e33affc0894643a92139ac6623b7a8: Generated new TSK 0
I20260620 14:24:42.266844 21498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:42.274276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 21406
W20260620 14:24:42.280004 21391 connection.cc:535] client connection to 127.0.0.1:44749 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44749): Connection reset by peer (error 104)
2026-06-20T14:24:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:42 dist-test-slave-87nc krb5kdc[21505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:42 dist-test-slave-87nc krb5kdc[21505](info): set up 2 sockets
Jun 20 14:24:42 dist-test-slave-87nc krb5kdc[21505](info): commencing operation
krb5kdc: starting...
W20260620 14:24:43.312155 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:43 dist-test-slave-87nc krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:43.435410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:43.544502 21518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:43.544819 21518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:43.544898 21518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:43.548842 21518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:43.548943 21518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:43.548981 21518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:43.549041 21518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:43.549079 21518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:43.554453 21518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:43.555557 21518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:43.556602 21518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:43.564975 21518 server_base.cc:1034] running on GCE node
W20260620 14:24:43.564908 21524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:43.564949 21526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:43.564908 21523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:43.565728 21518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:43.566720 21518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:43.567842 21518 hybrid_clock.cc:648] HybridClock initialized: now 1781965483567823 us; error 25 us; skew 500 ppm
Jun 20 14:24:43 dist-test-slave-87nc krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:43.573558 21518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:43.574424 21518 webserver.cc:460] Webserver started at http://127.0.0.1:35145/ using document root <none> and password file <none>
I20260620 14:24:43.574925 21518 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:43.574982 21518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:43.575142 21518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:43.577129 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "62b3f4c972094a0f9707425fba362e4e"
format_stamp: "Formatted at 2026-06-20 14:24:43 on dist-test-slave-87nc"
server_key: "45c1db9945a93b37f57dc80331fea5a9"
server_key_iv: "b42808eddc24a0d5813dfc3b120fcae2"
server_key_version: "encryptionkey@0"
I20260620 14:24:43.577630 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "62b3f4c972094a0f9707425fba362e4e"
format_stamp: "Formatted at 2026-06-20 14:24:43 on dist-test-slave-87nc"
server_key: "45c1db9945a93b37f57dc80331fea5a9"
server_key_iv: "b42808eddc24a0d5813dfc3b120fcae2"
server_key_version: "encryptionkey@0"
I20260620 14:24:43.581334 21518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:43.583742 21533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:43.585037 21518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:43.585165 21518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "62b3f4c972094a0f9707425fba362e4e"
format_stamp: "Formatted at 2026-06-20 14:24:43 on dist-test-slave-87nc"
server_key: "45c1db9945a93b37f57dc80331fea5a9"
server_key_iv: "b42808eddc24a0d5813dfc3b120fcae2"
server_key_version: "encryptionkey@0"
I20260620 14:24:43.585281 21518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:43.589866 21518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:43.593079 21518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:43.593274 21518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:43.600363 21518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37629
I20260620 14:24:43.600374 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37629 every 8 connection(s)
I20260620 14:24:43.601287 21518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:43.601612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 21518
I20260620 14:24:43.601755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:43.602093 392 external_mini_cluster.cc:1428] Setting key 6febf1b36f83111ddf57e2291bd48f83
I20260620 14:24:43.604490 21586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:43.607070 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.603394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:50043 (local address 127.0.0.1:37629)
0620 14:24:43.603806 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:43.603830 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:43.603866 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:24:43.604513 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.604555 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.604857 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.605131 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:43.605908 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:43.605913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:43.605924 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:43.605933 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:43.605981 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:43.606000 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:43.606105 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.606214 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.606415 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
I20260620 14:24:43.609728 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:43.610620 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e: Bootstrap starting.
I20260620 14:24:43.613415 21586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:43.613577 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.611874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:44771 (local address 127.0.0.1:37629)
0620 14:24:43.612012 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:43.612017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:43.612251 (+ 234us) server_negotiation.cc:407] Connection header received
0620 14:24:43.612353 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.612355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.612406 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.612467 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:43.613137 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:43.613139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:43.613141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:43.613150 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:43.613172 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:43.613181 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:43.613245 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.613373 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.613425 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:24:43.614257 21586 log.cc:824] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:43.616127 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e: No bootstrap required, opened a new log
I20260620 14:24:43.619467 21586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b3f4c972094a0f9707425fba362e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37629 } }
I20260620 14:24:43.619727 21586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:43.619829 21586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b3f4c972094a0f9707425fba362e4e, State: Initialized, Role: FOLLOWER
I20260620 14:24:43.620268 21586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b3f4c972094a0f9707425fba362e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37629 } }
I20260620 14:24:43.620420 21586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:43.620509 21586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:43.620609 21586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:43.621534 21586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b3f4c972094a0f9707425fba362e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37629 } }
I20260620 14:24:43.621877 21586 leader_election.cc:302] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62b3f4c972094a0f9707425fba362e4e; no voters:
I20260620 14:24:43.622233 21586 leader_election.cc:288] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:43.622298 21596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:43.622568 21596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [term 1 LEADER]: Becoming Leader. State: Replica: 62b3f4c972094a0f9707425fba362e4e, State: Running, Role: LEADER
I20260620 14:24:43.622927 21596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b3f4c972094a0f9707425fba362e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37629 } }
I20260620 14:24:43.623519 21586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:43.624794 21598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62b3f4c972094a0f9707425fba362e4e. Latest consensus state: current_term: 1 leader_uuid: "62b3f4c972094a0f9707425fba362e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b3f4c972094a0f9707425fba362e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37629 } } }
I20260620 14:24:43.624943 21598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:43.625128 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62b3f4c972094a0f9707425fba362e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b3f4c972094a0f9707425fba362e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37629 } } }
I20260620 14:24:43.625283 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:43.625598 21607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:43.628554 21607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:43.634363 21607 catalog_manager.cc:1347] Generated new cluster ID: 9874ad65775d4203893ee421be1d6221
I20260620 14:24:43.634439 21607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:43.644378 21607 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:43.644726 21607 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:43.649998 21607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62b3f4c972094a0f9707425fba362e4e: Generated new TSK 0
I20260620 14:24:43.650723 21607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:43.657982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 21518
2026-06-20T14:24:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:43 dist-test-slave-87nc krb5kdc[21617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:43 dist-test-slave-87nc krb5kdc[21617](info): set up 2 sockets
Jun 20 14:24:43 dist-test-slave-87nc krb5kdc[21617](info): commencing operation
krb5kdc: starting...
W20260620 14:24:44.692318 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:44 dist-test-slave-87nc krb5kdc[21617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:44.810662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:44.923461 21630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:44.923831 21630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:44.923957 21630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:44.928002 21630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:44.928109 21630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:44.928148 21630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:44.928184 21630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:44.928220 21630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:44.933772 21630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:44.935004 21630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:44.936167 21630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:44.945367 21638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:44.945367 21635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:44.945370 21636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:44.945688 21630 server_base.cc:1034] running on GCE node
I20260620 14:24:44.946463 21630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:44.947602 21630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:44.948792 21630 hybrid_clock.cc:648] HybridClock initialized: now 1781965484948772 us; error 42 us; skew 500 ppm
Jun 20 14:24:44 dist-test-slave-87nc krb5kdc[21617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:44.951592 21630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:44.952582 21630 webserver.cc:460] Webserver started at http://127.0.0.1:43083/ using document root <none> and password file <none>
I20260620 14:24:44.953192 21630 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:44.953292 21630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:44.953534 21630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:44.955744 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "aa67420c82f64ababdddd99e7f330fe0"
format_stamp: "Formatted at 2026-06-20 14:24:44 on dist-test-slave-87nc"
server_key: "d3c57f2a9231dccbadb4c9e7abfaad7b"
server_key_iv: "c3705a6da9e2aefac5c7297ce566e732"
server_key_version: "encryptionkey@0"
I20260620 14:24:44.956432 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa67420c82f64ababdddd99e7f330fe0"
format_stamp: "Formatted at 2026-06-20 14:24:44 on dist-test-slave-87nc"
server_key: "d3c57f2a9231dccbadb4c9e7abfaad7b"
server_key_iv: "c3705a6da9e2aefac5c7297ce566e732"
server_key_version: "encryptionkey@0"
I20260620 14:24:44.960445 21630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:24:44.963132 21645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:44.964461 21630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:44.964627 21630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "aa67420c82f64ababdddd99e7f330fe0"
format_stamp: "Formatted at 2026-06-20 14:24:44 on dist-test-slave-87nc"
server_key: "d3c57f2a9231dccbadb4c9e7abfaad7b"
server_key_iv: "c3705a6da9e2aefac5c7297ce566e732"
server_key_version: "encryptionkey@0"
I20260620 14:24:44.964766 21630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:44.994220 21630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:44.997905 21630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:44.998140 21630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:45.005550 21630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37275
I20260620 14:24:45.005575 21697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37275 every 8 connection(s)
I20260620 14:24:45.006560 21630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:45.007392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 21630
I20260620 14:24:45.007527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:45.007833 392 external_mini_cluster.cc:1428] Setting key f9ef5500b81bf6e1879ee3cd81d08751
I20260620 14:24:45.009894 21698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:45.016837 21698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0: Bootstrap starting.
Jun 20 14:24:45 dist-test-slave-87nc krb5kdc[21617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:24:45.019327 21698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:45.020181 21698 log.cc:824] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:45.022276 21698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0: No bootstrap required, opened a new log
I20260620 14:24:45.023370 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.009314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:49419 (local address 127.0.0.1:37275)
0620 14:24:45.009729 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:45.009747 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:45.009793 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:24:45.010663 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:45.010686 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:45.011125 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:45.011513 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:45.012282 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:45.013813 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:45.014749 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:45.015455 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:45.015665 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:45.018423 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:45.018429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:45.018442 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:45.018471 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:45.020407 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:45.020977 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:45.020987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:45.020992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:45.021110 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:45.021460 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:45.021468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:45.021469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:45.021855 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:45.022096 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:45.022414 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 14:24:45.022647 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260620 14:24:45.025928 21698 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa67420c82f64ababdddd99e7f330fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37275 } }
I20260620 14:24:45.026192 21698 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:45.026276 21698 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa67420c82f64ababdddd99e7f330fe0, State: Initialized, Role: FOLLOWER
I20260620 14:24:45.026636 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:45.026785 21698 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa67420c82f64ababdddd99e7f330fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37275 } }
I20260620 14:24:45.026980 21698 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:45.027051 21698 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:45.027179 21698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:45.028379 21698 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa67420c82f64ababdddd99e7f330fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37275 } }
I20260620 14:24:45.028684 21698 leader_election.cc:302] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa67420c82f64ababdddd99e7f330fe0; no voters:
I20260620 14:24:45.029034 21698 leader_election.cc:288] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:45.029333 21707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:45.029584 21707 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [term 1 LEADER]: Becoming Leader. State: Replica: aa67420c82f64ababdddd99e7f330fe0, State: Running, Role: LEADER
I20260620 14:24:45.029929 21707 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa67420c82f64ababdddd99e7f330fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37275 } }
I20260620 14:24:45.030301 21698 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:45.031256 21707 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa67420c82f64ababdddd99e7f330fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa67420c82f64ababdddd99e7f330fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37275 } } }
I20260620 14:24:45.031374 21707 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:45.031587 21707 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa67420c82f64ababdddd99e7f330fe0. Latest consensus state: current_term: 1 leader_uuid: "aa67420c82f64ababdddd99e7f330fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa67420c82f64ababdddd99e7f330fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37275 } } }
I20260620 14:24:45.031688 21707 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:45.031908 21718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:45.035238 21718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:45.038494 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.029079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:36827 (local address 127.0.0.1:37275)
0620 14:24:45.029204 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:45.029210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:45.029430 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:24:45.029639 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:45.029645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:45.029696 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:45.029746 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:45.030750 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:45.031539 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:45.032794 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:45.033128 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:45.033245 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:45.034909 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:45.034915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:45.034918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:45.034946 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:45.036541 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:45.037054 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:45.037058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:45.037059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:45.037173 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:45.037567 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:45.037571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:45.037572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:45.037784 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:45.037934 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:45.038234 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:24:45.038333 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:24:45.041464 21718 catalog_manager.cc:1347] Generated new cluster ID: 095a4323876f4708a4715dce6ef20539
I20260620 14:24:45.041567 21718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:45.065510 21718 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:45.066820 21718 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:45.077922 21718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa67420c82f64ababdddd99e7f330fe0: Generated new TSK 0
I20260620 14:24:45.078770 21718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:45.088058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 21630
W20260620 14:24:45.093324 21615 connection.cc:535] client connection to 127.0.0.1:37275 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37275): Connection reset by peer (error 104)
2026-06-20T14:24:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:45 dist-test-slave-87nc krb5kdc[21729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:45 dist-test-slave-87nc krb5kdc[21729](info): set up 2 sockets
Jun 20 14:24:45 dist-test-slave-87nc krb5kdc[21729](info): commencing operation
krb5kdc: starting...
W20260620 14:24:46.119943 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:46 dist-test-slave-87nc krb5kdc[21729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:46.240957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:46.351518 21742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:46.351814 21742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:46.351876 21742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:46.355738 21742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:46.355811 21742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:46.355831 21742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:46.355847 21742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:46.355862 21742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:46.361079 21742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32961
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:46.361968 21742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:46.362882 21742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:46.370934 21748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:46.371006 21747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:46.371122 21742 server_base.cc:1034] running on GCE node
W20260620 14:24:46.370929 21750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:46.371856 21742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:46.372908 21742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:46.374068 21742 hybrid_clock.cc:648] HybridClock initialized: now 1781965486374048 us; error 31 us; skew 500 ppm
Jun 20 14:24:46 dist-test-slave-87nc krb5kdc[21729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:46.376906 21742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:46.377743 21742 webserver.cc:460] Webserver started at http://127.0.0.1:42959/ using document root <none> and password file <none>
I20260620 14:24:46.378249 21742 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:46.378305 21742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:46.378464 21742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:46.380483 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "49509c3f21e544f086d9e3e74e008451"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-87nc"
server_key: "bd9ef7f66730f9dc5afb50f27353f662"
server_key_iv: "fcb870d588591302b8fdaa8c108808f0"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.380963 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "49509c3f21e544f086d9e3e74e008451"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-87nc"
server_key: "bd9ef7f66730f9dc5afb50f27353f662"
server_key_iv: "fcb870d588591302b8fdaa8c108808f0"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.384574 21742 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:46.386952 21757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:46.388361 21742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:46.388512 21742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "49509c3f21e544f086d9e3e74e008451"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-87nc"
server_key: "bd9ef7f66730f9dc5afb50f27353f662"
server_key_iv: "fcb870d588591302b8fdaa8c108808f0"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.388633 21742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:46.406831 21742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:46.410292 21742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:46.410506 21742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:46.417421 21742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32961
I20260620 14:24:46.417435 21809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32961 every 8 connection(s)
I20260620 14:24:46.418388 21742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:46.421128 21810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:46.426527 21810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451: Bootstrap starting.
I20260620 14:24:46.427551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 21742
I20260620 14:24:46.427742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:46.428042 392 external_mini_cluster.cc:1428] Setting key 97b4dddc4d1ad3f670d17ad85979dc48
I20260620 14:24:46.429328 21810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:46.430090 21810 log.cc:824] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:46.432262 21810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451: No bootstrap required, opened a new log
I20260620 14:24:46.436619 21810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49509c3f21e544f086d9e3e74e008451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32961 } }
I20260620 14:24:46.436885 21810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:46.436976 21810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49509c3f21e544f086d9e3e74e008451, State: Initialized, Role: FOLLOWER
I20260620 14:24:46.437419 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.429142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:40397 (local address 127.0.0.1:32961)
0620 14:24:46.429561 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.429577 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.429614 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:24:46.430398 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.430418 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.430880 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.431194 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.431518 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.432621 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.433825 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.434323 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.434453 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.435400 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.435406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.435420 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:46.435891 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.435949 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:46.435973 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.436075 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.436314 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.436468 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":136,"threads_started":1}
I20260620 14:24:46.437413 21810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49509c3f21e544f086d9e3e74e008451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32961 } }
I20260620 14:24:46.437575 21810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:46.437649 21810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:46.437770 21810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:46.438829 21810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49509c3f21e544f086d9e3e74e008451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32961 } }
I20260620 14:24:46.439169 21810 leader_election.cc:302] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49509c3f21e544f086d9e3e74e008451; no voters:
I20260620 14:24:46.439548 21810 leader_election.cc:288] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:46.439682 21815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:46.439944 21815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [term 1 LEADER]: Becoming Leader. State: Replica: 49509c3f21e544f086d9e3e74e008451, State: Running, Role: LEADER
I20260620 14:24:46.440275 21815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49509c3f21e544f086d9e3e74e008451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32961 } }
I20260620 14:24:46.440460 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:46.440864 21810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:46.441438 21815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49509c3f21e544f086d9e3e74e008451. Latest consensus state: current_term: 1 leader_uuid: "49509c3f21e544f086d9e3e74e008451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49509c3f21e544f086d9e3e74e008451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32961 } } }
I20260620 14:24:46.441556 21815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:46.442824 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49509c3f21e544f086d9e3e74e008451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49509c3f21e544f086d9e3e74e008451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32961 } } }
I20260620 14:24:46.442970 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451 [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:46.446281 21835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:46.446401 21835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:46.446537 21822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:46.447505 21822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:46.447662 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.443162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:57651 (local address 127.0.0.1:32961)
0620 14:24:46.443278 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.443285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.444034 (+ 749us) server_negotiation.cc:407] Connection header received
0620 14:24:46.444163 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.444165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.444204 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.444261 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.444557 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.445057 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.445904 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.446151 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.446262 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.447144 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.447148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.447151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:46.447164 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.447200 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:46.447215 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.447309 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.447501 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.447534 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:24:46.452939 21822 catalog_manager.cc:1347] Generated new cluster ID: 401e3990b25b483ea27da7fbc9ab6ee5
I20260620 14:24:46.453016 21822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:46.471225 21822 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:46.472265 21822 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:46.486038 21822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49509c3f21e544f086d9e3e74e008451: Generated new TSK 0
I20260620 14:24:46.486773 21822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:46.498471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 21742
W20260620 14:24:46.504076 21727 connection.cc:535] client connection to 127.0.0.1:32961 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32961): Connection reset by peer (error 104)
2026-06-20T14:24:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:46 dist-test-slave-87nc krb5kdc[21841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:46 dist-test-slave-87nc krb5kdc[21841](info): set up 2 sockets
Jun 20 14:24:46 dist-test-slave-87nc krb5kdc[21841](info): commencing operation
krb5kdc: starting...
W20260620 14:24:47.560025 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:47 dist-test-slave-87nc krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:24:47.676273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:47.785131 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:47.785408 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:47.785504 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:47.789496 21854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:47.789573 21854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:47.789609 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:47.789665 21854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:47.789702 21854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:47.794947 21854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:47.796015 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:47.797084 21854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:47.805110 21862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:47.805123 21860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:47.805182 21854 server_base.cc:1034] running on GCE node
W20260620 14:24:47.805120 21859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:47.806074 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:47.807547 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:47.808709 21854 hybrid_clock.cc:648] HybridClock initialized: now 1781965487808682 us; error 43 us; skew 500 ppm
Jun 20 14:24:47 dist-test-slave-87nc krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:47.811383 21854 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:24:47.812253 21854 webserver.cc:460] Webserver started at http://127.0.0.1:40523/ using document root <none> and password file <none>
I20260620 14:24:47.812757 21854 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:47.812812 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:47.812966 21854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:47.814908 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "05309af8e0504487aa25679ed805a177"
format_stamp: "Formatted at 2026-06-20 14:24:47 on dist-test-slave-87nc"
server_key: "1c61a8b5c912cb452686821a747c8a3e"
server_key_iv: "838181dd338cad6c9a83b326228b2e44"
server_key_version: "encryptionkey@0"
I20260620 14:24:47.815421 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "05309af8e0504487aa25679ed805a177"
format_stamp: "Formatted at 2026-06-20 14:24:47 on dist-test-slave-87nc"
server_key: "1c61a8b5c912cb452686821a747c8a3e"
server_key_iv: "838181dd338cad6c9a83b326228b2e44"
server_key_version: "encryptionkey@0"
I20260620 14:24:47.818882 21854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:24:47.821292 21869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:47.822446 21854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:47.822540 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "05309af8e0504487aa25679ed805a177"
format_stamp: "Formatted at 2026-06-20 14:24:47 on dist-test-slave-87nc"
server_key: "1c61a8b5c912cb452686821a747c8a3e"
server_key_iv: "838181dd338cad6c9a83b326228b2e44"
server_key_version: "encryptionkey@0"
I20260620 14:24:47.822624 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:47.826844 21854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:47.829555 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:47.829686 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:47.836169 21854 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41151
I20260620 14:24:47.836189 21921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41151 every 8 connection(s)
I20260620 14:24:47.837055 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:47.839699 21922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.841638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 21854
I20260620 14:24:47.841773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:47.842020 392 external_mini_cluster.cc:1428] Setting key 364b829fe338e16f0caca8305e56a014
I20260620 14:24:47.846083 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177: Bootstrap starting.
I20260620 14:24:47.847045 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.843329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:60227 (local address 127.0.0.1:41151)
0620 14:24:47.843757 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.843825 (+ 68us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.843869 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:24:47.844479 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.844494 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.844768 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.845094 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:47.845859 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:47.845864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:47.845876 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:47.845886 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:47.845936 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:47.845955 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:47.846105 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.846174 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.846338 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":135,"threads_started":1}
I20260620 14:24:47.848587 21922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.849488 21922 log.cc:824] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:47.849906 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:47.851825 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177: No bootstrap required, opened a new log
I20260620 14:24:47.853650 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.852066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:37323 (local address 127.0.0.1:41151)
0620 14:24:47.852204 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.852209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.852431 (+ 222us) server_negotiation.cc:407] Connection header received
0620 14:24:47.852518 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.852520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.852553 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.852628 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:47.853302 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:47.853306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:47.853308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:47.853322 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:47.853349 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:47.853361 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:47.853451 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.853500 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.853535 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:24:47.855753 21922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05309af8e0504487aa25679ed805a177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260620 14:24:47.856046 21922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.856089 21922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05309af8e0504487aa25679ed805a177, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.856441 21922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05309af8e0504487aa25679ed805a177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260620 14:24:47.856550 21922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:47.856592 21922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:47.856658 21922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.857470 21922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05309af8e0504487aa25679ed805a177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260620 14:24:47.857797 21922 leader_election.cc:302] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05309af8e0504487aa25679ed805a177; no voters:
I20260620 14:24:47.858086 21922 leader_election.cc:288] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:47.858273 21932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:47.858505 21932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [term 1 LEADER]: Becoming Leader. State: Replica: 05309af8e0504487aa25679ed805a177, State: Running, Role: LEADER
I20260620 14:24:47.858867 21932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05309af8e0504487aa25679ed805a177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260620 14:24:47.859365 21922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:47.860613 21933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05309af8e0504487aa25679ed805a177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05309af8e0504487aa25679ed805a177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } } }
I20260620 14:24:47.860733 21933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:47.861135 21941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:47.861115 21934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05309af8e0504487aa25679ed805a177. Latest consensus state: current_term: 1 leader_uuid: "05309af8e0504487aa25679ed805a177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05309af8e0504487aa25679ed805a177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } } }
I20260620 14:24:47.862092 21934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:47.864284 21941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:47.869408 21941 catalog_manager.cc:1347] Generated new cluster ID: 6a27987aa61c443c8c37b56011bd9164
I20260620 14:24:47.869477 21941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:47.887833 21941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:47.888144 21941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:47.893864 21941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05309af8e0504487aa25679ed805a177: Generated new TSK 0
I20260620 14:24:47.894665 21941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:47.898900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 21854
2026-06-20T14:24:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:47 dist-test-slave-87nc krb5kdc[21953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:47 dist-test-slave-87nc krb5kdc[21953](info): set up 2 sockets
Jun 20 14:24:47 dist-test-slave-87nc krb5kdc[21953](info): commencing operation
krb5kdc: starting...
W20260620 14:24:48.931958 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:49 dist-test-slave-87nc krb5kdc[21953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:49.050530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33951
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:49.156855 21966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:49.157179 21966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:49.157258 21966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:49.161254 21966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:49.161355 21966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:49.161393 21966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:49.161443 21966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:49.161475 21966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:49.166658 21966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:49.167817 21966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:49.168877 21966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:49.178125 21972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:49.178202 21971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:49.178185 21966 server_base.cc:1034] running on GCE node
W20260620 14:24:49.178141 21974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:49.179018 21966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:49.180334 21966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:49.181589 21966 hybrid_clock.cc:648] HybridClock initialized: now 1781965489181566 us; error 36 us; skew 500 ppm
Jun 20 14:24:49 dist-test-slave-87nc krb5kdc[21953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:49.187680 21966 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:49.188699 21966 webserver.cc:460] Webserver started at http://127.0.98.62:37879/ using document root <none> and password file <none>
I20260620 14:24:49.189241 21966 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:49.189301 21966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:49.189463 21966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:49.191460 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "fa26ee5178cf46338c0bd9da7800c5fc"
format_stamp: "Formatted at 2026-06-20 14:24:49 on dist-test-slave-87nc"
server_key: "d80a566e34e6175e2391e5058b56318a"
server_key_iv: "47a04383232d3ac68c72d1404e0c66ce"
server_key_version: "encryptionkey@0"
I20260620 14:24:49.192034 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa26ee5178cf46338c0bd9da7800c5fc"
format_stamp: "Formatted at 2026-06-20 14:24:49 on dist-test-slave-87nc"
server_key: "d80a566e34e6175e2391e5058b56318a"
server_key_iv: "47a04383232d3ac68c72d1404e0c66ce"
server_key_version: "encryptionkey@0"
I20260620 14:24:49.195672 21966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:24:49.198088 21981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:49.199297 21966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:49.199435 21966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "fa26ee5178cf46338c0bd9da7800c5fc"
format_stamp: "Formatted at 2026-06-20 14:24:49 on dist-test-slave-87nc"
server_key: "d80a566e34e6175e2391e5058b56318a"
server_key_iv: "47a04383232d3ac68c72d1404e0c66ce"
server_key_version: "encryptionkey@0"
I20260620 14:24:49.199550 21966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:49.216851 21966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:49.219748 21966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:49.219970 21966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:49.227049 21966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33951
I20260620 14:24:49.227066 22033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33951 every 8 connection(s)
I20260620 14:24:49.228022 21966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:49.230677 22034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:49.235956 22034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc: Bootstrap starting.
I20260620 14:24:49.237116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 21966
I20260620 14:24:49.237263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:49.237560 392 external_mini_cluster.cc:1428] Setting key f2207c441ecc3d7409bbcf2fa17c1ba0
I20260620 14:24:49.238773 22034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:49.239512 22034 log.cc:824] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:49.241348 22034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc: No bootstrap required, opened a new log
I20260620 14:24:49.244692 22034 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa26ee5178cf46338c0bd9da7800c5fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33951 } }
I20260620 14:24:49.244927 22034 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:49.245018 22034 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa26ee5178cf46338c0bd9da7800c5fc, State: Initialized, Role: FOLLOWER
I20260620 14:24:49.245440 22034 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa26ee5178cf46338c0bd9da7800c5fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33951 } }
I20260620 14:24:49.245616 22034 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:49.245702 22034 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:49.245833 22034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:24:49 dist-test-slave-87nc krb5kdc[21953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:49.246737 22034 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa26ee5178cf46338c0bd9da7800c5fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33951 } }
I20260620 14:24:49.247087 22034 leader_election.cc:302] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa26ee5178cf46338c0bd9da7800c5fc; no voters:
I20260620 14:24:49.247431 22034 leader_election.cc:288] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:49.247516 22039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:49.247749 22039 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [term 1 LEADER]: Becoming Leader. State: Replica: fa26ee5178cf46338c0bd9da7800c5fc, State: Running, Role: LEADER
I20260620 14:24:49.248137 22039 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa26ee5178cf46338c0bd9da7800c5fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33951 } }
I20260620 14:24:49.248674 22034 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:49.250021 22040 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa26ee5178cf46338c0bd9da7800c5fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa26ee5178cf46338c0bd9da7800c5fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33951 } } }
I20260620 14:24:49.250185 22040 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:49.251066 22050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:49.251968 22041 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa26ee5178cf46338c0bd9da7800c5fc. Latest consensus state: current_term: 1 leader_uuid: "fa26ee5178cf46338c0bd9da7800c5fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa26ee5178cf46338c0bd9da7800c5fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33951 } } }
I20260620 14:24:49.252063 22041 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:49.252801 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:49.238689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39425 (local address 127.0.98.62:33951)
0620 14:24:49.239109 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:49.239123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:49.239157 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:24:49.239912 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:49.239931 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:49.240251 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:49.240581 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:49.241375 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:49.242372 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:49.243217 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:49.243692 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:49.243902 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:49.246757 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:49.246763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:49.246775 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:49.246800 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:49.248887 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:49.249531 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:49.249536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:49.249542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:49.249677 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:49.250095 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:49.250099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:49.250100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:49.250613 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:49.250825 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:49.251603 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 14:24:49.251860 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260620 14:24:49.254012 22050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:49.256071 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:49.259680 22050 catalog_manager.cc:1347] Generated new cluster ID: 0e32d4c375d34e30855c212e185a987e
I20260620 14:24:49.259747 22050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:49.266194 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:49.258166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45351 (local address 127.0.98.62:33951)
0620 14:24:49.258262 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:49.258266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:49.258564 (+ 298us) server_negotiation.cc:407] Connection header received
0620 14:24:49.258702 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:49.258705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:49.258746 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:49.258850 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:49.259845 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:49.260476 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:49.261253 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:49.261521 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:49.261638 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:49.262989 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:49.262992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:49.262995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:49.263019 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:49.264453 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:49.264914 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:49.264917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:49.264919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:49.264996 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:49.265311 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:49.265314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:49.265316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:49.265495 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:49.265664 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:49.265924 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:24:49.266016 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:24:49.273497 22050 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:49.274690 22050 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:49.301998 22050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa26ee5178cf46338c0bd9da7800c5fc: Generated new TSK 0
I20260620 14:24:49.302979 22050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:49.310871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 21966
W20260620 14:24:49.316169 21951 connection.cc:535] client connection to 127.0.98.62:33951 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33951): Connection reset by peer (error 104)
2026-06-20T14:24:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:49 dist-test-slave-87nc krb5kdc[22065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:49 dist-test-slave-87nc krb5kdc[22065](info): set up 2 sockets
Jun 20 14:24:49 dist-test-slave-87nc krb5kdc[22065](info): commencing operation
krb5kdc: starting...
W20260620 14:24:50.340376 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:50 dist-test-slave-87nc krb5kdc[22065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:50.459935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44751
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:50.575347 22078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:50.575647 22078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:50.575765 22078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:50.579685 22078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:50.579754 22078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:50.579851 22078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:50.579900 22078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:50.579959 22078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:50.585337 22078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:50.586405 22078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:50.587484 22078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:50.596228 22086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.596280 22083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.596247 22084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:50.596510 22078 server_base.cc:1034] running on GCE node
I20260620 14:24:50.597200 22078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:50.598317 22078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:50.599574 22078 hybrid_clock.cc:648] HybridClock initialized: now 1781965490599542 us; error 47 us; skew 500 ppm
Jun 20 14:24:50 dist-test-slave-87nc krb5kdc[22065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:50.602521 22078 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:50.603509 22078 webserver.cc:460] Webserver started at http://127.0.98.62:41571/ using document root <none> and password file <none>
I20260620 14:24:50.604166 22078 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:50.604233 22078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:50.604453 22078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:50.606611 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "e176c71db18b4d4b8bdb99bea028c5a2"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-87nc"
server_key: "f28223885ed4151ba402a5034c4d79ec"
server_key_iv: "ad67686c6bab9be3e766db2b04a75f8b"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.607188 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "e176c71db18b4d4b8bdb99bea028c5a2"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-87nc"
server_key: "f28223885ed4151ba402a5034c4d79ec"
server_key_iv: "ad67686c6bab9be3e766db2b04a75f8b"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.611090 22078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 14:24:50.613600 22093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.614845 22078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:24:50.614990 22078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "e176c71db18b4d4b8bdb99bea028c5a2"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-87nc"
server_key: "f28223885ed4151ba402a5034c4d79ec"
server_key_iv: "ad67686c6bab9be3e766db2b04a75f8b"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.615094 22078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:50.633613 22078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:50.636858 22078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:50.637025 22078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:50.644567 22078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44751
I20260620 14:24:50.644661 22145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44751 every 8 connection(s)
I20260620 14:24:50.645695 22078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:50.646423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 22078
I20260620 14:24:50.646580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:50.646868 392 external_mini_cluster.cc:1428] Setting key d8a809a274fe3f318e288f29666753c6
I20260620 14:24:50.648911 22146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:50.655488 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2: Bootstrap starting.
I20260620 14:24:50.656810 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:50.648211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50305 (local address 127.0.98.62:44751)
0620 14:24:50.648618 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:50.648638 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:50.648679 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:24:50.649501 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:50.649522 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:50.649931 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:50.650274 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:50.650698 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.652263 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.653155 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.653834 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.653987 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:50.654849 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:50.654855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:50.654869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:50.655330 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:50.655405 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:50.655435 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:50.655570 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:50.655788 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:24:50.655914 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260620 14:24:50.658100 22146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:50.658931 22146 log.cc:824] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:50.659829 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:50.661260 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2: No bootstrap required, opened a new log
I20260620 14:24:50.665149 22146 raft_consensus.cc:348] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176c71db18b4d4b8bdb99bea028c5a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44751 } }
I20260620 14:24:50.665445 22146 raft_consensus.cc:374] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:50.665517 22146 raft_consensus.cc:729] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e176c71db18b4d4b8bdb99bea028c5a2, State: Initialized, Role: FOLLOWER
I20260620 14:24:50.665997 22146 consensus_queue.cc:260] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176c71db18b4d4b8bdb99bea028c5a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44751 } }
I20260620 14:24:50.666155 22146 raft_consensus.cc:388] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:50.666230 22146 raft_consensus.cc:482] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:50.666383 22146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:50.667588 22146 raft_consensus.cc:504] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176c71db18b4d4b8bdb99bea028c5a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44751 } }
I20260620 14:24:50.668035 22146 leader_election.cc:302] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e176c71db18b4d4b8bdb99bea028c5a2; no voters:
I20260620 14:24:50.668241 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:50.661929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51825 (local address 127.0.98.62:44751)
0620 14:24:50.662046 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:50.662050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:50.664086 (+ 2036us) server_negotiation.cc:407] Connection header received
0620 14:24:50.664161 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:50.664163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:50.664203 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:50.664291 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:50.664620 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.665137 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.666316 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.666612 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.666750 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:50.667663 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:50.667666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:50.667668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:50.667680 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:50.667705 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:50.667749 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:50.667868 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:50.668064 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:24:50.668103 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:24:50.668442 22146 leader_election.cc:288] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:50.668568 22156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:50.668831 22156 raft_consensus.cc:686] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [term 1 LEADER]: Becoming Leader. State: Replica: e176c71db18b4d4b8bdb99bea028c5a2, State: Running, Role: LEADER
I20260620 14:24:50.669189 22156 consensus_queue.cc:237] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176c71db18b4d4b8bdb99bea028c5a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44751 } }
I20260620 14:24:50.669786 22146 sys_catalog.cc:564] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:50.670462 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e176c71db18b4d4b8bdb99bea028c5a2. Latest consensus state: current_term: 1 leader_uuid: "e176c71db18b4d4b8bdb99bea028c5a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176c71db18b4d4b8bdb99bea028c5a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44751 } } }
I20260620 14:24:50.670687 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:50.670864 22158 sys_catalog.cc:455] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e176c71db18b4d4b8bdb99bea028c5a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e176c71db18b4d4b8bdb99bea028c5a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44751 } } }
I20260620 14:24:50.670960 22158 sys_catalog.cc:458] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:50.671118 22165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:50.674365 22165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:50.680092 22165 catalog_manager.cc:1347] Generated new cluster ID: 56419ffdc2bd4089881dfcaa310c7906
I20260620 14:24:50.680166 22165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:50.721117 22165 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:50.722435 22165 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:50.727587 22165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e176c71db18b4d4b8bdb99bea028c5a2: Generated new TSK 0
I20260620 14:24:50.728300 22165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:50.734707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 22078
W20260620 14:24:50.739995 22063 connection.cc:535] client connection to 127.0.98.62:44751 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44751): Connection reset by peer (error 104)
2026-06-20T14:24:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:50 dist-test-slave-87nc krb5kdc[22177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:50 dist-test-slave-87nc krb5kdc[22177](info): set up 2 sockets
Jun 20 14:24:50 dist-test-slave-87nc krb5kdc[22177](info): commencing operation
krb5kdc: starting...
W20260620 14:24:51.768059 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:51 dist-test-slave-87nc krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:51.882834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:51.993192 22190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:51.993482 22190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:51.993526 22190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:51.997587 22190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:51.997640 22190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:51.997721 22190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:51.997748 22190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:51.997773 22190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:52.003741 22190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:52.004796 22190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:52.005755 22190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:52.014386 22198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:52.014401 22195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:52.014573 22190 server_base.cc:1034] running on GCE node
W20260620 14:24:52.014401 22196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:52.015312 22190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:52.016469 22190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:52.017627 22190 hybrid_clock.cc:648] HybridClock initialized: now 1781965492017595 us; error 46 us; skew 500 ppm
Jun 20 14:24:52 dist-test-slave-87nc krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:52.020308 22190 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:52.021206 22190 webserver.cc:460] Webserver started at http://127.0.98.62:44209/ using document root <none> and password file <none>
I20260620 14:24:52.021741 22190 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:52.021806 22190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:52.022045 22190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:52.024119 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "1be103c67cfb4e839d7ebc80b42aeb38"
format_stamp: "Formatted at 2026-06-20 14:24:52 on dist-test-slave-87nc"
server_key: "56935d3f8506639530765ab01e4cc330"
server_key_iv: "740dd1402bec35bde3eb396024ee3008"
server_key_version: "encryptionkey@0"
I20260620 14:24:52.024682 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "1be103c67cfb4e839d7ebc80b42aeb38"
format_stamp: "Formatted at 2026-06-20 14:24:52 on dist-test-slave-87nc"
server_key: "56935d3f8506639530765ab01e4cc330"
server_key_iv: "740dd1402bec35bde3eb396024ee3008"
server_key_version: "encryptionkey@0"
I20260620 14:24:52.028259 22190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:24:52.030584 22205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:52.031754 22190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:52.031980 22190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "1be103c67cfb4e839d7ebc80b42aeb38"
format_stamp: "Formatted at 2026-06-20 14:24:52 on dist-test-slave-87nc"
server_key: "56935d3f8506639530765ab01e4cc330"
server_key_iv: "740dd1402bec35bde3eb396024ee3008"
server_key_version: "encryptionkey@0"
I20260620 14:24:52.032073 22190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:52.036490 22190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:52.039168 22190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:52.039351 22190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:52.046243 22190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44489
I20260620 14:24:52.046253 22257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44489 every 8 connection(s)
I20260620 14:24:52.047114 22190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:52.048626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 22190
I20260620 14:24:52.048765 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:52.049050 392 external_mini_cluster.cc:1428] Setting key 7cb97715af2c49bf1a5c709a3466e91a
I20260620 14:24:52.050037 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:52.054014 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:52.050387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57903 (local address 127.0.98.62:44489)
0620 14:24:52.050795 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:52.050812 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:52.050852 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:24:52.051452 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:52.051467 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:52.051722 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:52.052050 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:52.052868 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:52.052873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:52.052884 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:52.052893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:52.052944 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:52.052962 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:52.053105 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:52.053198 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:24:52.053352 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":139,"threads_started":1}
I20260620 14:24:52.056067 22258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38: Bootstrap starting.
I20260620 14:24:52.056727 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:52.058708 22258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:52.059587 22258 log.cc:824] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:52.060489 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:52.058899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55097 (local address 127.0.98.62:44489)
0620 14:24:52.059030 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:52.059034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:52.059207 (+ 173us) server_negotiation.cc:407] Connection header received
0620 14:24:52.059311 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:52.059313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:52.059338 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:52.059399 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:52.060141 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:52.060145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:52.060148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:52.060162 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:52.060188 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:52.060203 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:52.060300 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:52.060334 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:24:52.060369 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:24:52.061683 22258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38: No bootstrap required, opened a new log
I20260620 14:24:52.065117 22258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be103c67cfb4e839d7ebc80b42aeb38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44489 } }
I20260620 14:24:52.065335 22258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:52.065428 22258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be103c67cfb4e839d7ebc80b42aeb38, State: Initialized, Role: FOLLOWER
I20260620 14:24:52.065856 22258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be103c67cfb4e839d7ebc80b42aeb38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44489 } }
I20260620 14:24:52.066007 22258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:52.066075 22258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:52.066171 22258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:52.066998 22258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be103c67cfb4e839d7ebc80b42aeb38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44489 } }
I20260620 14:24:52.067327 22258 leader_election.cc:302] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be103c67cfb4e839d7ebc80b42aeb38; no voters:
I20260620 14:24:52.067653 22258 leader_election.cc:288] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:52.067752 22268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:52.068022 22268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 1be103c67cfb4e839d7ebc80b42aeb38, State: Running, Role: LEADER
I20260620 14:24:52.068377 22268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be103c67cfb4e839d7ebc80b42aeb38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44489 } }
I20260620 14:24:52.068802 22258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:52.070080 22270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be103c67cfb4e839d7ebc80b42aeb38. Latest consensus state: current_term: 1 leader_uuid: "1be103c67cfb4e839d7ebc80b42aeb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be103c67cfb4e839d7ebc80b42aeb38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44489 } } }
I20260620 14:24:52.070094 22269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1be103c67cfb4e839d7ebc80b42aeb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be103c67cfb4e839d7ebc80b42aeb38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44489 } } }
I20260620 14:24:52.070204 22269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:52.070204 22270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:52.070637 22277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:52.073241 22277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:52.078075 22277 catalog_manager.cc:1347] Generated new cluster ID: eac2c69d2f064c5a98470f9686388e1d
I20260620 14:24:52.078178 22277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:52.111928 22277 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:52.112303 22277 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:52.120251 22277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1be103c67cfb4e839d7ebc80b42aeb38: Generated new TSK 0
I20260620 14:24:52.121027 22277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:52.130935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 22190
2026-06-20T14:24:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:52 dist-test-slave-87nc krb5kdc[22289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:52 dist-test-slave-87nc krb5kdc[22289](info): set up 2 sockets
Jun 20 14:24:52 dist-test-slave-87nc krb5kdc[22289](info): commencing operation
krb5kdc: starting...
W20260620 14:24:53.224130 392 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:53 dist-test-slave-87nc krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:53.342887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:53.452538 22302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:53.452841 22302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:53.452884 22302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:53.456881 22302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:53.456950 22302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:53.456969 22302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:53.456986 22302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:53.457002 22302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:53.462365 22302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:53.463459 22302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:53.464514 22302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:53.473441 22310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:53.473541 22307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:53.473529 22302 server_base.cc:1034] running on GCE node
W20260620 14:24:53.473446 22308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:53.474359 22302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:53.475478 22302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:53.476673 22302 hybrid_clock.cc:648] HybridClock initialized: now 1781965493476629 us; error 60 us; skew 500 ppm
Jun 20 14:24:53 dist-test-slave-87nc krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:53.479487 22302 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:53.480424 22302 webserver.cc:460] Webserver started at http://127.0.98.62:35243/ using document root <none> and password file <none>
I20260620 14:24:53.480945 22302 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:53.481002 22302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:53.481158 22302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:53.483107 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "602958dc54a747589e93a6742e1a94a6"
format_stamp: "Formatted at 2026-06-20 14:24:53 on dist-test-slave-87nc"
server_key: "74767565d6fb5c97f2b805d2b69deffe"
server_key_iv: "9e21c2dab4aa1467bf8628dd58cf4a2f"
server_key_version: "encryptionkey@0"
I20260620 14:24:53.483659 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "602958dc54a747589e93a6742e1a94a6"
format_stamp: "Formatted at 2026-06-20 14:24:53 on dist-test-slave-87nc"
server_key: "74767565d6fb5c97f2b805d2b69deffe"
server_key_iv: "9e21c2dab4aa1467bf8628dd58cf4a2f"
server_key_version: "encryptionkey@0"
I20260620 14:24:53.487409 22302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:24:53.489897 22317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:53.491118 22302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:24:53.491251 22302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "602958dc54a747589e93a6742e1a94a6"
format_stamp: "Formatted at 2026-06-20 14:24:53 on dist-test-slave-87nc"
server_key: "74767565d6fb5c97f2b805d2b69deffe"
server_key_iv: "9e21c2dab4aa1467bf8628dd58cf4a2f"
server_key_version: "encryptionkey@0"
I20260620 14:24:53.491359 22302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:53.503145 22302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:53.506505 22302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:53.506714 22302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:53.513863 22302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35527
I20260620 14:24:53.513875 22369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35527 every 8 connection(s)
I20260620 14:24:53.514797 22302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:53.517594 22370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:53.518972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 22302
I20260620 14:24:53.519145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:53.519425 392 external_mini_cluster.cc:1428] Setting key 5e5c5f4ffcd176bdd8922ff89cb7c5d4
I20260620 14:24:53.524104 22370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6: Bootstrap starting.
I20260620 14:24:53.526676 22370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:53.527711 22370 log.cc:824] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:24:53 dist-test-slave-87nc krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:53.529978 22370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6: No bootstrap required, opened a new log
I20260620 14:24:53.534381 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:53.520804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48839 (local address 127.0.98.62:35527)
0620 14:24:53.521208 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:53.521223 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:53.521257 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:24:53.521871 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:53.521890 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:53.522233 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:53.522505 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:53.523433 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.524513 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.525665 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.526111 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.526219 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:53.529124 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:53.529133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:53.529151 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:53.529199 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:53.531176 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:53.531596 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:53.531602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:53.531608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:53.531741 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:53.532045 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:53.532051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:53.532055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:53.532614 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:53.532804 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:53.533128 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:24:53.533412 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
I20260620 14:24:53.534596 22370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602958dc54a747589e93a6742e1a94a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35527 } }
I20260620 14:24:53.534870 22370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:53.534943 22370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602958dc54a747589e93a6742e1a94a6, State: Initialized, Role: FOLLOWER
I20260620 14:24:53.535338 22370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602958dc54a747589e93a6742e1a94a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35527 } }
I20260620 14:24:53.535495 22370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:53.535565 22370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:53.535658 22370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:53.536783 22370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602958dc54a747589e93a6742e1a94a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35527 } }
I20260620 14:24:53.537149 22370 leader_election.cc:302] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 602958dc54a747589e93a6742e1a94a6; no voters:
I20260620 14:24:53.537523 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:53.537520 22370 leader_election.cc:288] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:53.537855 22376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:53.538075 22376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [term 1 LEADER]: Becoming Leader. State: Replica: 602958dc54a747589e93a6742e1a94a6, State: Running, Role: LEADER
I20260620 14:24:53.538419 22376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602958dc54a747589e93a6742e1a94a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35527 } }
I20260620 14:24:53.538816 22370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:53.540096 22377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "602958dc54a747589e93a6742e1a94a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602958dc54a747589e93a6742e1a94a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35527 } } }
I20260620 14:24:53.540222 22377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:53.540576 22375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 602958dc54a747589e93a6742e1a94a6. Latest consensus state: current_term: 1 leader_uuid: "602958dc54a747589e93a6742e1a94a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602958dc54a747589e93a6742e1a94a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35527 } } }
I20260620 14:24:53.540678 22375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6 [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:53.543800 22395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:53.543942 22395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:53.544027 22391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:53.544867 22391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:53.548580 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:53.539629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34263 (local address 127.0.98.62:35527)
0620 14:24:53.539748 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:53.539754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:53.540148 (+ 394us) server_negotiation.cc:407] Connection header received
0620 14:24:53.540287 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:53.540291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:53.540339 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:53.540397 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:53.541432 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.542179 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.543149 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.543450 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.543576 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:53.545097 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:53.545101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:53.545103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:53.545137 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:53.546619 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:53.547110 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:53.547115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:53.547116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:53.547210 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:53.547530 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:53.547533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:53.547535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:53.547853 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:53.548008 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:53.548271 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 14:24:53.548401 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:24:53.551249 22391 catalog_manager.cc:1347] Generated new cluster ID: 9ee9fa75098049438cfff195a12d6184
I20260620 14:24:53.551321 22391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:53.578006 22391 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:53.579260 22391 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:53.607890 22391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 602958dc54a747589e93a6742e1a94a6: Generated new TSK 0
I20260620 14:24:53.608860 22391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:53.613968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 22302
2026-06-20T14:24:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:53 dist-test-slave-87nc krb5kdc[22401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:53 dist-test-slave-87nc krb5kdc[22401](info): set up 2 sockets
Jun 20 14:24:53 dist-test-slave-87nc krb5kdc[22401](info): commencing operation
krb5kdc: starting...
W20260620 14:24:54.660022 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:54 dist-test-slave-87nc krb5kdc[22401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:54.778542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41479
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:54.892809 22414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:54.893189 22414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:54.893307 22414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:54.897464 22414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:54.897573 22414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:54.897619 22414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:54.897665 22414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:54.897704 22414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:54.903216 22414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:54.904412 22414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:54.905496 22414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:54.915310 22420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:54.915318 22419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:54.915318 22422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:54.915566 22414 server_base.cc:1034] running on GCE node
I20260620 14:24:54.916311 22414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:54.917405 22414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:54.918579 22414 hybrid_clock.cc:648] HybridClock initialized: now 1781965494918559 us; error 35 us; skew 500 ppm
Jun 20 14:24:54 dist-test-slave-87nc krb5kdc[22401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:54.925908 22414 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:54.926898 22414 webserver.cc:460] Webserver started at http://127.0.98.62:43005/ using document root <none> and password file <none>
I20260620 14:24:54.927484 22414 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:54.927583 22414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:54.927882 22414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:54.930058 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "f2218b3e718a4d0f921137d670ca8a7e"
format_stamp: "Formatted at 2026-06-20 14:24:54 on dist-test-slave-87nc"
server_key: "93b99fca643af5be5085e31989123a0c"
server_key_iv: "b4ecca0c9c3e48a14aa941eebb358fe9"
server_key_version: "encryptionkey@0"
I20260620 14:24:54.930683 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2218b3e718a4d0f921137d670ca8a7e"
format_stamp: "Formatted at 2026-06-20 14:24:54 on dist-test-slave-87nc"
server_key: "93b99fca643af5be5085e31989123a0c"
server_key_iv: "b4ecca0c9c3e48a14aa941eebb358fe9"
server_key_version: "encryptionkey@0"
I20260620 14:24:54.934718 22414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:24:54.937312 22429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:54.938802 22414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:24:54.938946 22414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "f2218b3e718a4d0f921137d670ca8a7e"
format_stamp: "Formatted at 2026-06-20 14:24:54 on dist-test-slave-87nc"
server_key: "93b99fca643af5be5085e31989123a0c"
server_key_iv: "b4ecca0c9c3e48a14aa941eebb358fe9"
server_key_version: "encryptionkey@0"
I20260620 14:24:54.939075 22414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:54.967608 22414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:54.970680 22414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:54.970901 22414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:54.978174 22414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41479
I20260620 14:24:54.978188 22481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41479 every 8 connection(s)
I20260620 14:24:54.979179 22414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:54.982066 22482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:54.985025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 22414
I20260620 14:24:54.985193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:54.985455 392 external_mini_cluster.cc:1428] Setting key b993b5e04e10df947aafc933a3381026
I20260620 14:24:54.988255 22482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e: Bootstrap starting.
I20260620 14:24:54.991040 22482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:54.991959 22482 log.cc:824] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:54.994274 22482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e: No bootstrap required, opened a new log
I20260620 14:24:54.995785 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:54.986911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59389 (local address 127.0.98.62:41479)
0620 14:24:54.987340 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:54.987364 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:54.987408 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:24:54.988254 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:54.988279 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:54.988722 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:54.989104 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:54.989527 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.990975 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.991987 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.992634 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.992801 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:54.993660 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:54.993666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:54.993683 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:54.994233 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:54.994304 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:54.994338 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:54.994435 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:54.994602 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:24:54.994826 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
I20260620 14:24:54.998627 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:54.998744 22482 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2218b3e718a4d0f921137d670ca8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41479 } }
I20260620 14:24:54.998986 22482 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:54.999056 22482 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2218b3e718a4d0f921137d670ca8a7e, State: Initialized, Role: FOLLOWER
I20260620 14:24:54.999482 22482 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2218b3e718a4d0f921137d670ca8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41479 } }
I20260620 14:24:54.999609 22482 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:54.999693 22482 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:54.999825 22482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:55.000847 22482 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2218b3e718a4d0f921137d670ca8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41479 } }
I20260620 14:24:55.001237 22482 leader_election.cc:302] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2218b3e718a4d0f921137d670ca8a7e; no voters:
I20260620 14:24:55.001602 22482 leader_election.cc:288] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:55.001722 22492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:55.001957 22492 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [term 1 LEADER]: Becoming Leader. State: Replica: f2218b3e718a4d0f921137d670ca8a7e, State: Running, Role: LEADER
I20260620 14:24:55.002308 22492 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2218b3e718a4d0f921137d670ca8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41479 } }
I20260620 14:24:55.002921 22482 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:55.003931 22494 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2218b3e718a4d0f921137d670ca8a7e. Latest consensus state: current_term: 1 leader_uuid: "f2218b3e718a4d0f921137d670ca8a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2218b3e718a4d0f921137d670ca8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41479 } } }
I20260620 14:24:55.004068 22494 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:55.004367 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2218b3e718a4d0f921137d670ca8a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2218b3e718a4d0f921137d670ca8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41479 } } }
I20260620 14:24:55.004451 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:55.004842 22501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:55.005676 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:55.000789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60815 (local address 127.0.98.62:41479)
0620 14:24:55.000921 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:55.000926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:55.001075 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:24:55.001168 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:55.001171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:55.001218 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:55.001300 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:55.001587 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.002336 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.003448 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.003860 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.003964 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:55.004977 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:55.004984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:55.004990 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:55.005002 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:55.005032 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:55.005046 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:55.005129 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:55.005455 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:24:55.005542 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:24:55.008605 22501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:55.013810 22501 catalog_manager.cc:1347] Generated new cluster ID: 41cdeec1eb7f4fc6b0db1741978dd5aa
I20260620 14:24:55.013881 22501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:55.030534 22501 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:55.031893 22501 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:55.044551 22501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2218b3e718a4d0f921137d670ca8a7e: Generated new TSK 0
I20260620 14:24:55.045418 22501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:55.055439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 22414
2026-06-20T14:24:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:55 dist-test-slave-87nc krb5kdc[22513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:55 dist-test-slave-87nc krb5kdc[22513](info): set up 2 sockets
Jun 20 14:24:55 dist-test-slave-87nc krb5kdc[22513](info): commencing operation
krb5kdc: starting...
W20260620 14:24:56.092139 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:56 dist-test-slave-87nc krb5kdc[22513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:56.210702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36345
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:56.321545 22526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:56.321849 22526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:56.321894 22526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:56.326033 22526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:56.326103 22526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:56.326130 22526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:56.326148 22526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:56.326162 22526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:56.331485 22526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:56.332608 22526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:56.333578 22526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:56.342103 22532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:56.342119 22531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:56.342142 22534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:56.342562 22526 server_base.cc:1034] running on GCE node
I20260620 14:24:56.343189 22526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:56.344369 22526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:56.345533 22526 hybrid_clock.cc:648] HybridClock initialized: now 1781965496345504 us; error 41 us; skew 500 ppm
Jun 20 14:24:56 dist-test-slave-87nc krb5kdc[22513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:56.348424 22526 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:56.349391 22526 webserver.cc:460] Webserver started at http://127.0.98.62:33563/ using document root <none> and password file <none>
I20260620 14:24:56.349958 22526 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:56.350054 22526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:56.350299 22526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:56.352479 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "82d3733fedd64752b9a019fe65ce0e85"
format_stamp: "Formatted at 2026-06-20 14:24:56 on dist-test-slave-87nc"
server_key: "0cd5f079102e916e752dcd2a5143f6d9"
server_key_iv: "2f2689e29c8ed5dfd3e6507e0386e78f"
server_key_version: "encryptionkey@0"
I20260620 14:24:56.353051 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "82d3733fedd64752b9a019fe65ce0e85"
format_stamp: "Formatted at 2026-06-20 14:24:56 on dist-test-slave-87nc"
server_key: "0cd5f079102e916e752dcd2a5143f6d9"
server_key_iv: "2f2689e29c8ed5dfd3e6507e0386e78f"
server_key_version: "encryptionkey@0"
I20260620 14:24:56.356834 22526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:24:56.359350 22541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:56.360687 22526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:56.360831 22526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "82d3733fedd64752b9a019fe65ce0e85"
format_stamp: "Formatted at 2026-06-20 14:24:56 on dist-test-slave-87nc"
server_key: "0cd5f079102e916e752dcd2a5143f6d9"
server_key_iv: "2f2689e29c8ed5dfd3e6507e0386e78f"
server_key_version: "encryptionkey@0"
I20260620 14:24:56.360948 22526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:56.365566 22526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:56.368800 22526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:56.368996 22526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:56.376233 22526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36345
I20260620 14:24:56.376252 22593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36345 every 8 connection(s)
I20260620 14:24:56.377213 22526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:56.379920 22594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:56.385178 22594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85: Bootstrap starting.
I20260620 14:24:56.386828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 22526
I20260620 14:24:56.386969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:56.387269 392 external_mini_cluster.cc:1428] Setting key 26ffda533a04bb445f07e7007b69dcf3
I20260620 14:24:56.387979 22594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:56.388871 22594 log.cc:824] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:56.390863 22594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85: No bootstrap required, opened a new log
I20260620 14:24:56.392477 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:56.388439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48111 (local address 127.0.98.62:36345)
0620 14:24:56.388854 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:56.388867 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:56.388901 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:24:56.389767 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:56.389788 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:56.390066 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:56.390423 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:56.391258 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:56.391262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:56.391274 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:56.391285 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:56.391335 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:56.391354 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:56.391471 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:56.391571 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:24:56.391733 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":117,"threads_started":1}
I20260620 14:24:56.394878 22594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d3733fedd64752b9a019fe65ce0e85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36345 } }
I20260620 14:24:56.395175 22594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:56.395241 22594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d3733fedd64752b9a019fe65ce0e85, State: Initialized, Role: FOLLOWER
I20260620 14:24:56.395453 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:56.395700 22594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d3733fedd64752b9a019fe65ce0e85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36345 } }
I20260620 14:24:56.395897 22594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:56.395982 22594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:56.396080 22594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:56.397111 22594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d3733fedd64752b9a019fe65ce0e85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36345 } }
I20260620 14:24:56.397446 22594 leader_election.cc:302] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d3733fedd64752b9a019fe65ce0e85; no voters:
I20260620 14:24:56.397826 22594 leader_election.cc:288] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:56.397878 22604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:56.398131 22604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [term 1 LEADER]: Becoming Leader. State: Replica: 82d3733fedd64752b9a019fe65ce0e85, State: Running, Role: LEADER
I20260620 14:24:56.398499 22604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d3733fedd64752b9a019fe65ce0e85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36345 } }
I20260620 14:24:56.399205 22594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:56.399507 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:56.397586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59221 (local address 127.0.98.62:36345)
0620 14:24:56.397708 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:56.397712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:56.397929 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:24:56.398109 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:56.398112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:56.398155 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:56.398236 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:56.399066 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:56.399069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:56.399072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:56.399083 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:56.399119 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:56.399133 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:56.399220 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:56.399361 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 14:24:56.399398 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:24:56.399798 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d3733fedd64752b9a019fe65ce0e85. Latest consensus state: current_term: 1 leader_uuid: "82d3733fedd64752b9a019fe65ce0e85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d3733fedd64752b9a019fe65ce0e85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36345 } } }
I20260620 14:24:56.399907 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:56.400231 22606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d3733fedd64752b9a019fe65ce0e85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d3733fedd64752b9a019fe65ce0e85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36345 } } }
I20260620 14:24:56.400306 22606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:56.400368 22609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:56.403124 22609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:56.408156 22609 catalog_manager.cc:1347] Generated new cluster ID: c643eb8dd6a4470c8a438bb6d0ce3bd2
I20260620 14:24:56.408226 22609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:56.414331 22609 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:56.414664 22609 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:56.430608 22609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d3733fedd64752b9a019fe65ce0e85: Generated new TSK 0
I20260620 14:24:56.431483 22609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:56.435434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 22526
2026-06-20T14:24:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1379 ms)
[----------] 18 tests from AuthTokenIssuingTest (25533 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:56 dist-test-slave-87nc krb5kdc[22625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:56 dist-test-slave-87nc krb5kdc[22625](info): set up 2 sockets
Jun 20 14:24:56 dist-test-slave-87nc krb5kdc[22625](info): commencing operation
krb5kdc: starting...
W20260620 14:24:57.500255 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:57 dist-test-slave-87nc krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:57.618777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33473
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33473,127.0.98.61:33659,127.0.98.60:34701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:57.730340 22638 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:24:57.730773 22638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:57.730840 22638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:57.730923 22638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:57.735129 22638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:57.735224 22638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:57.735260 22638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:57.735297 22638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:57.735334 22638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:57.740645 22638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33473,127.0.98.61:33659,127.0.98.60:34701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:57.741753 22638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:57.742794 22638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:57.751173 22646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:57.751159 22644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:57.751369 22638 server_base.cc:1034] running on GCE node
W20260620 14:24:57.751161 22643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:57.752121 22638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:57.753157 22638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:57.754323 22638 hybrid_clock.cc:648] HybridClock initialized: now 1781965497754306 us; error 31 us; skew 500 ppm
Jun 20 14:24:57 dist-test-slave-87nc krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:57.757031 22638 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:57.757974 22638 webserver.cc:460] Webserver started at http://127.0.98.62:45567/ using document root <none> and password file <none>
I20260620 14:24:57.758540 22638 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:57.758600 22638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:57.758836 22638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:57.761341 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "9cc2f3ccaa894196afaae09262462a4a"
format_stamp: "Formatted at 2026-06-20 14:24:57 on dist-test-slave-87nc"
server_key: "17a9cdd2ac0679694e5e4b78037d74a6"
server_key_iv: "1f463a4a0b0abcf14d196bc6bf8d594b"
server_key_version: "encryptionkey@0"
I20260620 14:24:57.761998 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "9cc2f3ccaa894196afaae09262462a4a"
format_stamp: "Formatted at 2026-06-20 14:24:57 on dist-test-slave-87nc"
server_key: "17a9cdd2ac0679694e5e4b78037d74a6"
server_key_iv: "1f463a4a0b0abcf14d196bc6bf8d594b"
server_key_version: "encryptionkey@0"
I20260620 14:24:57.765806 22638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:24:57.768261 22653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:57.769464 22638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:57.769600 22638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "9cc2f3ccaa894196afaae09262462a4a"
format_stamp: "Formatted at 2026-06-20 14:24:57 on dist-test-slave-87nc"
server_key: "17a9cdd2ac0679694e5e4b78037d74a6"
server_key_iv: "1f463a4a0b0abcf14d196bc6bf8d594b"
server_key_version: "encryptionkey@0"
I20260620 14:24:57.769709 22638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:57.788496 22638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:57.791397 22638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:57.791613 22638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:57.798808 22638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33473
I20260620 14:24:57.799098 22705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33473 every 8 connection(s)
I20260620 14:24:57.799858 22638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:57.802604 22706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:57.805437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 22638
I20260620 14:24:57.805583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:57.805846 392 external_mini_cluster.cc:1428] Setting key 3d83e7f8862c53436474615229575e8c
I20260620 14:24:57.806838 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 14:24:57 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:57.822093 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.809412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33473 (local address 127.0.0.1:42826)
0620 14:24:57.809954 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.809988 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:57.810812 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:57.811458 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:57.811463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:57.811934 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:57.812160 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:57.812174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.813300 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.813303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:57.814057 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:57.814066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.814697 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.814701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:57.814806 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:57.815390 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:57.815406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:57.816930 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:57.818966 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:57.818978 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:57.818991 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:57.819328 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:57.819655 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:57.819658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:57.819661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:57.819840 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:57.820430 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:57.820440 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:57.820637 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 14:24:57.820839 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 14:24:57.821268 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":82,"spinlock_wait_cycles":12160,"thread_start_us":160,"threads_started":1}
I20260620 14:24:57.822093 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.809411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.0.98.62:33473)
0620 14:24:57.809809 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:57.809828 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:57.810093 (+ 265us) server_negotiation.cc:407] Connection header received
0620 14:24:57.811103 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:57.811107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:57.811280 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:57.811462 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:57.812300 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.813148 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.814186 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.814539 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.814705 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:57.817083 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:57.817092 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:57.817108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:57.817140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:57.818855 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:57.819448 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:57.819453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:57.819454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:57.819555 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:57.819932 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:57.819935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:57.819936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:57.820306 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:57.820410 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:57.820909 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 14:24:57.821154 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20260620 14:24:57.824592 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:24:57.825361 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.825054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33659 (local address 127.0.0.1:34090)
0620 14:24:57.825238 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.825302 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 14:24:57.825590 22657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
I20260620 14:24:57.826445 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.826257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33659 (local address 127.0.0.1:34096)
0620 14:24:57.826353 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.826400 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:24:57.826574 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33659: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:24:57.831019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33659
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33473,127.0.98.61:33659,127.0.98.60:34701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:24:57.875792 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } attempt: 1
I20260620 14:24:57.876675 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.876325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33659 (local address 127.0.0.1:34108)
0620 14:24:57.876500 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.876577 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:24:57.877439 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.877207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33659 (local address 127.0.0.1:34118)
0620 14:24:57.877316 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.877369 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:24:57.877588 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33659: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
W20260620 14:24:57.944180 22713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:24:57.944502 22713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:57.944552 22713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:57.944615 22713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:24:57.945802 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } attempt: 2
I20260620 14:24:57.946667 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.946326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33659 (local address 127.0.0.1:34124)
0620 14:24:57.946492 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.946568 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:24:57.947451 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.947221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33659 (local address 127.0.0.1:34132)
0620 14:24:57.947332 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.947385 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:24:57.947615 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33659: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33659: connect: Connection refused (error 111)
W20260620 14:24:57.949267 22713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:57.949347 22713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:57.949394 22713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:57.949432 22713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:57.949467 22713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:57.954806 22713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33473,127.0.98.61:33659,127.0.98.60:34701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:57.955999 22713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:57.957043 22713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:57.965612 22718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:57.965612 22719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:57.965849 22713 server_base.cc:1034] running on GCE node
W20260620 14:24:57.965631 22721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:57.966441 22713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:57.967595 22713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:57.968794 22713 hybrid_clock.cc:648] HybridClock initialized: now 1781965497968777 us; error 40 us; skew 500 ppm
Jun 20 14:24:57 dist-test-slave-87nc krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:57.971465 22713 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:24:57.972409 22713 webserver.cc:460] Webserver started at http://127.0.98.61:34467/ using document root <none> and password file <none>
I20260620 14:24:57.973047 22713 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:57.973137 22713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:57.973368 22713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:57.975426 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/data/instance:
uuid: "12c35dd69f05418f98b5c74c3fd30abd"
format_stamp: "Formatted at 2026-06-20 14:24:57 on dist-test-slave-87nc"
server_key: "49155260c2a425ef07d4565653731694"
server_key_iv: "d82fd1d29d9dcbc648421eb1e90fb60a"
server_key_version: "encryptionkey@0"
I20260620 14:24:57.976009 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/wal/instance:
uuid: "12c35dd69f05418f98b5c74c3fd30abd"
format_stamp: "Formatted at 2026-06-20 14:24:57 on dist-test-slave-87nc"
server_key: "49155260c2a425ef07d4565653731694"
server_key_iv: "d82fd1d29d9dcbc648421eb1e90fb60a"
server_key_version: "encryptionkey@0"
I20260620 14:24:57.979672 22713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:24:57.982125 22728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:57.983290 22713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:57.983439 22713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/wal
uuid: "12c35dd69f05418f98b5c74c3fd30abd"
format_stamp: "Formatted at 2026-06-20 14:24:57 on dist-test-slave-87nc"
server_key: "49155260c2a425ef07d4565653731694"
server_key_iv: "d82fd1d29d9dcbc648421eb1e90fb60a"
server_key_version: "encryptionkey@0"
I20260620 14:24:57.983551 22713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:58.009914 22713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:58.013078 22713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:58.013305 22713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:58.020522 22713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33659
I20260620 14:24:58.020803 22780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33659 every 8 connection(s)
I20260620 14:24:58.021518 22713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:24:58.024325 22781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:58.027257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 22713
I20260620 14:24:58.027380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-1/wal/instance
I20260620 14:24:58.027638 392 external_mini_cluster.cc:1428] Setting key 633f784ae88e0fc52dfe7c7c79593cbe
I20260620 14:24:58.028329 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 14:24:58.038816 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } attempt: 3
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:58.045115 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.030563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.0.98.62:33473)
0620 14:24:58.030748 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.030753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.031545 (+ 792us) server_negotiation.cc:407] Connection header received
0620 14:24:58.032515 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.032519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.032582 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.032699 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.033818 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.034344 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.035457 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.035705 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.035839 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.040062 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.040069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.040076 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.040110 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.042716 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.043390 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.043394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.043395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.043481 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.043941 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.043944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.043946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.044182 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.044295 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.044825 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.044945 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:24:58.046008 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.030962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33473 (local address 127.0.0.1:42838)
0620 14:24:58.031399 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.031440 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.032291 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.032861 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.032868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.033225 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.033676 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.033689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.034471 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.034476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.035319 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.035328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.035872 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.035876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.036041 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.036682 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.036697 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.039873 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:58.042849 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.042858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.042877 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.043280 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.043589 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.043592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.043594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.043838 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.044296 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:58.044303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:58.044564 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 14:24:58.044757 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 14:24:58.045073 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:24:58.047402 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:24:58.051508 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.039328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33659 (local address 127.0.0.1:34134)
0620 14:24:58.039536 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.039554 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.039632 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.040283 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.040286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.040481 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.040613 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.040623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.043219 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.043224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.044027 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.044037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.044978 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.044985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.045126 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.045627 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.045655 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.047179 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:58.049861 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.049867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.049872 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.050134 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.050444 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.050447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.050449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.050538 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.050975 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:58.050979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:58.051076 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 14:24:58.051207 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:24:58.051339 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 14:24:58.051605 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.039455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.0.98.61:33659)
0620 14:24:58.039887 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.039897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.039921 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:24:58.039961 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.039968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.040153 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.040297 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.041858 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.043115 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.044142 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.044816 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.046985 (+ 2169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.047338 (+ 353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.047347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.047361 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.047385 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.049754 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.050252 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.050258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.050259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.050345 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.050615 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.050620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.050622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.050866 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.051010 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.051289 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.051418 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":339,"thread_start_us":89,"threads_started":1}
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:24:58.053555 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:24:58.054255 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.054003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:35942)
0620 14:24:58.054114 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.054169 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:24:58.055076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34701
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33473,127.0.98.61:33659,127.0.98.60:34701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:24:58.056622 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.054696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:35946)
0620 14:24:58.056490 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.056532 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1729}
W20260620 14:24:58.056770 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34701: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
I20260620 14:24:58.057593 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.048063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33659 (local address 127.0.0.1:34146)
0620 14:24:58.048168 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.048183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.048261 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.048705 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.048710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.048942 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.049072 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.049080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.050106 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.050110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.050676 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.050684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.051151 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.051155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.051303 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.051914 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.051933 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.053961 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:58.055982 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.055988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.055993 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.056266 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.056591 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.056594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.056596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.056688 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.057098 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:58.057101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:58.057181 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 14:24:58.057290 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:24:58.057423 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":40}
I20260620 14:24:58.057631 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.048087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.0.98.61:33659)
0620 14:24:58.048454 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.048463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.048479 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:24:58.048516 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.048522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.048572 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.048657 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.049249 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.050003 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.050794 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.051037 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.051153 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.054111 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.054117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.054123 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.054152 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.055882 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.056377 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.056384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.056387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.056484 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.056795 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.056798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.056799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.056996 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.057108 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.057341 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.057474 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
I20260620 14:24:58.058220 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:24:58.058857 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.058616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:35962)
0620 14:24:58.058736 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.058809 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:24:58.059093 22729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
I20260620 14:24:58.059998 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.059826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:35972)
0620 14:24:58.059912 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.059952 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:24:58.060236 22781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34701: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
I20260620 14:24:58.087940 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } attempt: 1
I20260620 14:24:58.088791 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.088500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:35982)
0620 14:24:58.088647 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.088716 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:24:58.089483 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.089273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:35998)
0620 14:24:58.089376 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.089436 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:24:58.089591 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34701: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
I20260620 14:24:58.109493 22781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } attempt: 1
I20260620 14:24:58.110328 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.110046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:36006)
0620 14:24:58.110190 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.110252 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 14:24:58.111059 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.110854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:36018)
0620 14:24:58.110956 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.111008 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:24:58.111186 22781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34701: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
I20260620 14:24:58.164737 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } attempt: 2
I20260620 14:24:58.165556 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.165269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:36026)
0620 14:24:58.165417 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.165482 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:24:58.166253 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.166043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:36036)
0620 14:24:58.166146 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.166206 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:24:58.166404 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34701: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
W20260620 14:24:58.167502 22788 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:24:58.167865 22788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:58.167928 22788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:58.168006 22788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:58.172180 22788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:58.172261 22788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:58.172308 22788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:58.172340 22788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:58.172394 22788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:58.177965 22788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33473,127.0.98.61:33659,127.0.98.60:34701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:24:58.179154 22788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:58.179345 22781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } attempt: 2
I20260620 14:24:58.180171 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.179897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:36052)
0620 14:24:58.180028 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.180097 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:24:58.180373 22788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:58.180934 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.180720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:36060)
0620 14:24:58.180839 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.180884 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:24:58.181064 22781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34701: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34701: connect: Connection refused (error 111)
W20260620 14:24:58.189152 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:58.189226 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:58.189368 22788 server_base.cc:1034] running on GCE node
W20260620 14:24:58.189226 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:58.190078 22788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:58.191143 22788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:58.192330 22788 hybrid_clock.cc:648] HybridClock initialized: now 1781965498192314 us; error 28 us; skew 500 ppm
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:58.195040 22788 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:24:58.195996 22788 webserver.cc:460] Webserver started at http://127.0.98.60:40609/ using document root <none> and password file <none>
I20260620 14:24:58.196544 22788 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:58.196633 22788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:58.196859 22788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:58.198859 22788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/data/instance:
uuid: "9a6a21e177b248548ac1cc71699979e0"
format_stamp: "Formatted at 2026-06-20 14:24:58 on dist-test-slave-87nc"
server_key: "66c6273f5101b54b3b46d9c9f1a00086"
server_key_iv: "807b333e0f919b98eabdb2c797881489"
server_key_version: "encryptionkey@0"
I20260620 14:24:58.199421 22788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/wal/instance:
uuid: "9a6a21e177b248548ac1cc71699979e0"
format_stamp: "Formatted at 2026-06-20 14:24:58 on dist-test-slave-87nc"
server_key: "66c6273f5101b54b3b46d9c9f1a00086"
server_key_iv: "807b333e0f919b98eabdb2c797881489"
server_key_version: "encryptionkey@0"
I20260620 14:24:58.202978 22788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:24:58.205389 22804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:58.206558 22788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:58.206693 22788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/wal
uuid: "9a6a21e177b248548ac1cc71699979e0"
format_stamp: "Formatted at 2026-06-20 14:24:58 on dist-test-slave-87nc"
server_key: "66c6273f5101b54b3b46d9c9f1a00086"
server_key_iv: "807b333e0f919b98eabdb2c797881489"
server_key_version: "encryptionkey@0"
I20260620 14:24:58.206802 22788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:58.228775 22788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:58.232156 22788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:58.232386 22788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:58.239289 22788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34701
I20260620 14:24:58.239568 22856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34701 every 8 connection(s)
I20260620 14:24:58.240274 22788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:24:58.241802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 22788
I20260620 14:24:58.241920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965214882667-392-0/minicluster-data/master-2/wal/instance
I20260620 14:24:58.242210 392 external_mini_cluster.cc:1428] Setting key 4cec0d157b2b9f61116cf3e3db8a2aac
I20260620 14:24:58.243332 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:58.247027 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:58.256858 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.243578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.0.98.60:34701)
0620 14:24:58.244045 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.244060 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.244094 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:24:58.244734 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.244750 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.245061 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.245396 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.246191 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.247159 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.248096 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.248724 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.248886 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.251863 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.251868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.251876 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.251901 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.253619 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.254112 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.254117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.254121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.254270 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.254547 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.254551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.254553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.254914 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.255111 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.255437 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.255734 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":159,"threads_started":1}
I20260620 14:24:58.257694 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.248695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33473 (local address 127.0.0.1:42844)
0620 14:24:58.249161 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.249216 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.249389 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.249736 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.249740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.250050 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.250169 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.250174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.251110 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.251113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.251872 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.251882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.252333 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.252336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.252451 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.252861 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.252873 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.254150 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:58.256177 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.256188 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.256211 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.256450 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.256724 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.256726 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.256727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.256794 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.257236 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:58.257240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:58.257331 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 14:24:58.257441 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:24:58.257564 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":129,"threads_started":1}
I20260620 14:24:58.257766 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.248761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.0.98.62:33473)
0620 14:24:58.248897 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.248902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.249296 (+ 394us) server_negotiation.cc:407] Connection header received
0620 14:24:58.249543 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.249547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.249604 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.249674 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.250312 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.250991 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.251994 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.252213 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.252322 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.254300 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.254308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.254311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.254334 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.256051 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.256541 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.256545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.256546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.256632 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.256883 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.256886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.256889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.257127 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.257242 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.257517 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.257628 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:24:58.258831 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:58.265551 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } attempt: 3
I20260620 14:24:58.268723 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.259351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33659 (local address 127.0.0.1:34160)
0620 14:24:58.259501 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.259516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.259583 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.259876 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.259878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.259999 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.260080 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.260084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.261031 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.261034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.261619 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.261628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.262089 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.262093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.262197 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.262874 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.262888 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.264409 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:58.266563 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.266565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.266566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.266893 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.267516 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.267519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.267521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.267618 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.268173 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:58.268177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:58.268276 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 14:24:58.268396 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:24:58.268538 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 14:24:58.268778 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.259470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.0.98.61:33659)
0620 14:24:58.259610 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.259613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.259626 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:24:58.259669 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.259671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.259718 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.259868 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.260225 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.260907 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.261743 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.261984 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.262068 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.264562 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.264565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.264567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.264614 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.266277 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.267063 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.267067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.267069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.267163 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.267726 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.267730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.267732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.268003 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.268125 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.268470 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.268608 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:24:58.270105 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:24:58.270309 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.259456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.0.98.62:33473)
0620 14:24:58.260003 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.260006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.260017 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:58.260046 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.260048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.260078 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.260141 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.260735 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.261340 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.262232 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.262597 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.262716 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.266386 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.266390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.266393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.266419 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.268110 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.268608 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.268612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.268613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.268699 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.269002 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.269005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.269007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.269242 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.269458 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.270006 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.270138 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:24:58.272230 22781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } attempt: 3
I20260620 14:24:58.275593 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.266034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:36092)
0620 14:24:58.266185 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.266201 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.266277 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.266512 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.266517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.266704 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.266808 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.266814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.268097 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.268101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.269031 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.269043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.269572 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.269576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.269660 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.270080 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.270094 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.271476 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:58.273819 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.273841 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.273843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.274131 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.274483 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.274487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.274489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.274588 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.275076 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:58.275079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:58.275185 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 14:24:58.275280 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:24:58.275416 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:24:58.276592 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.266091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.0.98.60:34701)
0620 14:24:58.266219 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.266223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.266239 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:24:58.266367 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.266369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.266406 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.266473 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.266955 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.267975 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.269152 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.269475 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.269567 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.271619 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.271624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.271626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.271656 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.273689 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.274262 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.274265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.274267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.274356 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.274710 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.274714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.274716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.274941 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.276168 (+ 1227us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.276248 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.276396 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:24:58 dist-test-slave-87nc krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:24:58.280689 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a: Bootstrap starting.
I20260620 14:24:58.282358 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.270584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:36098)
0620 14:24:58.270788 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.270801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.270875 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.272007 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.272011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.272216 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.272338 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.272349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.274208 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.274214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.275132 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.275142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.275889 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.275894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.275990 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.276504 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.276518 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.278578 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:58.280928 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.280931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.280933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.281214 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.281467 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.281469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.281471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.281537 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.281889 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:58.281892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:58.281962 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 14:24:58.282061 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:24:58.282213 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":28}
I20260620 14:24:58.282805 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.270657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.0.98.60:34701)
0620 14:24:58.271797 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.271807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.271820 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:24:58.271851 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.271854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.271901 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.271986 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.272476 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.274029 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.275248 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.275638 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.275748 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.278716 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.278719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.278722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.278744 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.280823 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.281308 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.281311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.281313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.281386 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.281611 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.281614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.281615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.281791 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.282481 (+ 690us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.282547 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.282644 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1047,"thread_start_us":130,"threads_started":1}
I20260620 14:24:58.284093 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.272578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34701 (local address 127.0.0.1:36104)
0620 14:24:58.272712 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.272729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.272818 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.273847 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.273850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.274014 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.274105 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.274111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.275343 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.275347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.276570 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.276582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.277313 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.277320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.277420 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.277830 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.277843 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.279281 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:58.282088 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.282095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.282099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.282410 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.283034 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:58.283040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:58.283043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:58.283134 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:58.283599 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:58.283602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:58.283695 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:24:58.283815 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:24:58.283943 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":3}
I20260620 14:24:58.284686 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.272825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.0.98.60:34701)
0620 14:24:58.273615 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.273622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.273633 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:58.273662 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.273665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.273702 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.274215 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.274442 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.275216 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.276719 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.277142 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.278421 (+ 1279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.279422 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.279426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.279429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.279454 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.281934 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.282842 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.282846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.282848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.282932 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.283267 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.283272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.283275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.283491 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.284291 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.284364 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.284478 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":730,"thread_start_us":106,"threads_started":1}
I20260620 14:24:58.284730 22706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:58.285663 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.271747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.0.98.61:33659)
0620 14:24:58.271916 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.271923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.271940 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:24:58.272369 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.272372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.272417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.272469 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.273478 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.274243 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.275343 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.275691 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.275805 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.280465 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.280471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.280477 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.280525 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.282702 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.283438 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.283445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.283447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.283550 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.284184 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.284190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.284191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.284398 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.284635 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.285377 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.285518 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":75}
I20260620 14:24:58.286293 22857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0: Bootstrap starting.
I20260620 14:24:58.287500 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd: Bootstrap starting.
I20260620 14:24:58.289077 22857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:58.289249 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:58.289757 22781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:58.290009 22857 log.cc:824] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:58.290386 22781 log.cc:824] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:58.291303 22706 log.cc:824] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:58.290620 22680 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9cc2f3ccaa894196afaae09262462a4a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42850
I20260620 14:24:58.293000 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd: No bootstrap required, opened a new log
I20260620 14:24:58.293340 22857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0: No bootstrap required, opened a new log
I20260620 14:24:58.297123 22781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } }
I20260620 14:24:58.297343 22781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c35dd69f05418f98b5c74c3fd30abd, State: Initialized, Role: FOLLOWER
I20260620 14:24:58.297412 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a: No bootstrap required, opened a new log
I20260620 14:24:58.297506 22857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } }
I20260620 14:24:58.297741 22857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6a21e177b248548ac1cc71699979e0, State: Initialized, Role: FOLLOWER
I20260620 14:24:58.297709 22781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } }
I20260620 14:24:58.298059 22857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } }
I20260620 14:24:58.298406 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.298699 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:58.299067 22781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:58.299325 22857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:58.300693 22706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } }
I20260620 14:24:58.300913 22706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc2f3ccaa894196afaae09262462a4a, State: Initialized, Role: FOLLOWER
I20260620 14:24:58.301265 22706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } }
I20260620 14:24:58.302276 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.303011 22706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:58.303171 22680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:24:58.303278 22680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:58.304852 22680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } }
I20260620 14:24:58.305845 22680 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [CANDIDATE]: Term 1 election: Requested vote from peers 12c35dd69f05418f98b5c74c3fd30abd (127.0.98.61:33659), 9a6a21e177b248548ac1cc71699979e0 (127.0.98.60:34701)
I20260620 14:24:58.302978 22788 master_runner.cc:186] Error getting master registration for 127.0.98.62:33473: OK, instance_id { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" instance_seqno: 1781965497796781 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:24:58.306352 22788 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:24:58.302964 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:58.306344 22755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cc2f3ccaa894196afaae09262462a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "12c35dd69f05418f98b5c74c3fd30abd"
I20260620 14:24:58.303107 22713 master_runner.cc:186] Error getting master registration for 127.0.98.62:33473: OK, instance_id { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" instance_seqno: 1781965497796781 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:24:58.306586 22755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:58.306638 22713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:24:58.307155 22831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cc2f3ccaa894196afaae09262462a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9a6a21e177b248548ac1cc71699979e0"
I20260620 14:24:58.307358 22831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:58.308008 22755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cc2f3ccaa894196afaae09262462a4a in term 1.
I20260620 14:24:58.308447 22657 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c35dd69f05418f98b5c74c3fd30abd, 9cc2f3ccaa894196afaae09262462a4a; no voters:
W20260620 14:24:58.308542 22890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:58.308615 22890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:58.308655 22831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cc2f3ccaa894196afaae09262462a4a in term 1.
I20260620 14:24:58.308930 22902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:58.309145 22902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [term 1 LEADER]: Becoming Leader. State: Replica: 9cc2f3ccaa894196afaae09262462a4a, State: Running, Role: LEADER
I20260620 14:24:58.309528 22902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } }
I20260620 14:24:58.311712 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc2f3ccaa894196afaae09262462a4a. Latest consensus state: current_term: 1 leader_uuid: "9cc2f3ccaa894196afaae09262462a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.311858 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:58.313248 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc2f3ccaa894196afaae09262462a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.313365 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:58.319527 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.309260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.0.98.61:33659)
0620 14:24:58.309364 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.309369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.309808 (+ 439us) server_negotiation.cc:407] Connection header received
0620 14:24:58.310022 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.310028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.310080 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.310133 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.311233 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.312089 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.313359 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.313703 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.313776 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.315735 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.315739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.315742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.315802 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.317765 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.318218 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.318222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.318223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.318313 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.318612 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.318615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.318617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.318827 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.318917 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.319256 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.319374 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:24:58.320243 22914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:58.323587 22914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 14:24:58.324563 22889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:58.324637 22889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:58.325585 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.309901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.0.98.62:33473)
0620 14:24:58.310010 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.310014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.311339 (+ 1325us) server_negotiation.cc:407] Connection header received
0620 14:24:58.311491 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.311494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.311537 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.311583 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.312501 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.313223 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.315276 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.315625 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.315704 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.321498 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.321502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.321505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.321530 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.323300 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.323931 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.323936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.323938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.324022 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.324335 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.324338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.324340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.324619 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.324772 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.325259 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.325409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:24:58.328557 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.310585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.0.98.60:34701)
0620 14:24:58.310793 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.310800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.314248 (+ 3448us) server_negotiation.cc:407] Connection header received
0620 14:24:58.314423 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.314429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.314491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.314545 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.317660 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.318665 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.321882 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.322295 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.322589 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.324309 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:58.324316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:58.324321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:58.324347 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:58.326569 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.327168 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.327172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.327173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.327257 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:58.327535 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:58.327538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:58.327540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:58.327721 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:58.327897 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:58.328274 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 14:24:58.328400 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 14:24:58.338238 22831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [term 1 FOLLOWER]: Refusing update from remote peer 9cc2f3ccaa894196afaae09262462a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:58.341342 22903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:58.345577 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc2f3ccaa894196afaae09262462a4a. Latest consensus state: current_term: 1 leader_uuid: "9cc2f3ccaa894196afaae09262462a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.345722 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:58.346881 22755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [term 1 FOLLOWER]: Refusing update from remote peer 9cc2f3ccaa894196afaae09262462a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:58.347563 22903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:58.352207 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc2f3ccaa894196afaae09262462a4a. Latest consensus state: current_term: 1 leader_uuid: "9cc2f3ccaa894196afaae09262462a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.352337 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:58.358471 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cc2f3ccaa894196afaae09262462a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.358611 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:58.358819 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cc2f3ccaa894196afaae09262462a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.358928 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:58.358927 22917 mvcc.cc:205] Tried to move back new op lower bound from 7298930681184997376 to 7298930681080242176. Current Snapshot: MvccSnapshot[applied={T|T < 7298930681184997376}]
I20260620 14:24:58.359402 22920 mvcc.cc:205] Tried to move back new op lower bound from 7298930681184997376 to 7298930681080242176. Current Snapshot: MvccSnapshot[applied={T|T < 7298930681184997376}]
I20260620 14:24:58.360770 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cc2f3ccaa894196afaae09262462a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.360881 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:58.362053 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cc2f3ccaa894196afaae09262462a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2f3ccaa894196afaae09262462a4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33473 } } peers { permanent_uuid: "12c35dd69f05418f98b5c74c3fd30abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33659 } } peers { permanent_uuid: "9a6a21e177b248548ac1cc71699979e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34701 } } }
I20260620 14:24:58.362169 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:58.362824 22918 mvcc.cc:205] Tried to move back new op lower bound from 7298930681184997376 to 7298930681080242176. Current Snapshot: MvccSnapshot[applied={T|T < 7298930681184997376}]
I20260620 14:24:58.364073 22914 catalog_manager.cc:1347] Generated new cluster ID: c8872f59647143c2a2b07647909d1e8d
I20260620 14:24:58.364137 22914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:58.412108 22914 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:58.413441 22914 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:58.426636 22914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc2f3ccaa894196afaae09262462a4a: Generated new TSK 0
I20260620 14:24:58.427548 22914 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:24:58.453469 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.450916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.0.98.61:33659)
0620 14:24:58.451059 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.451066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.452331 (+ 1265us) server_negotiation.cc:407] Connection header received
0620 14:24:58.452548 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.452552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.452604 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.452660 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.453315 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34188: BlockingRecv error: recv got EOF from 127.0.0.1:34188 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 14:24:58.455286 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.451686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.0.98.62:33473)
0620 14:24:58.451825 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.451829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.454222 (+ 2393us) server_negotiation.cc:407] Connection header received
0620 14:24:58.454465 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.454469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.454535 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.454589 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.455148 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42872: BlockingRecv error: recv got EOF from 127.0.0.1:42872 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 14:24:58.457753 22863 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.456039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.0.98.60:34701)
0620 14:24:58.456162 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.456167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.456700 (+ 533us) server_negotiation.cc:407] Connection header received
0620 14:24:58.456908 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.456911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.456968 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.457025 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.457618 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36124: BlockingRecv error: recv got EOF from 127.0.0.1:36124 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 14:24:58.462958 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.461254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.0.98.61:33659)
0620 14:24:58.461368 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.461372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.461920 (+ 548us) server_negotiation.cc:407] Connection header received
0620 14:24:58.462159 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.462163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.462240 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.462297 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.462850 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34202: BlockingRecv error: recv got EOF from 127.0.0.1:34202 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 14:24:58.468595 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.466804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.0.98.61:33659)
0620 14:24:58.466926 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.466931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.467529 (+ 598us) server_negotiation.cc:407] Connection header received
0620 14:24:58.467749 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.467753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.467842 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.467903 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.468472 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34210: BlockingRecv error: recv got EOF from 127.0.0.1:34210 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 14:24:58.474822 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.472964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.0.98.61:33659)
0620 14:24:58.473077 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.473082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.473794 (+ 712us) server_negotiation.cc:407] Connection header received
0620 14:24:58.474010 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.474014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.474073 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.474129 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.474702 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34222: BlockingRecv error: recv got EOF from 127.0.0.1:34222 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:24:58.483319 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.481354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.0.98.61:33659)
0620 14:24:58.481538 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.481544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.482161 (+ 617us) server_negotiation.cc:407] Connection header received
0620 14:24:58.482377 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.482381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.482460 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.482531 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.483167 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34224: BlockingRecv error: recv got EOF from 127.0.0.1:34224 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 14:24:58.496093 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.494042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.0.98.61:33659)
0620 14:24:58.494182 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.494187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.494902 (+ 715us) server_negotiation.cc:407] Connection header received
0620 14:24:58.495156 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.495161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.495227 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.495290 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.495967 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34226: BlockingRecv error: recv got EOF from 127.0.0.1:34226 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 14:24:58.517015 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.514968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.0.98.61:33659)
0620 14:24:58.515101 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.515106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.515864 (+ 758us) server_negotiation.cc:407] Connection header received
0620 14:24:58.516086 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.516090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.516150 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.516206 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.516846 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34234: BlockingRecv error: recv got EOF from 127.0.0.1:34234 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:24:58.552968 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.551614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.0.98.61:33659)
0620 14:24:58.551742 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.551746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.552021 (+ 275us) server_negotiation.cc:407] Connection header received
0620 14:24:58.552171 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.552174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.552223 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.552297 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.552843 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34236: BlockingRecv error: recv got EOF from 127.0.0.1:34236 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:24:58.620045 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.619060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.0.98.61:33659)
0620 14:24:58.619193 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.619197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.619300 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:24:58.619403 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.619406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.619453 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.619535 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.619950 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34250: BlockingRecv error: recv got EOF from 127.0.0.1:34250 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 14:24:58.751605 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.750414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.0.98.61:33659)
0620 14:24:58.750567 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.750571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.750708 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:24:58.750885 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.750889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.750945 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.751035 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.751489 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34252: BlockingRecv error: recv got EOF from 127.0.0.1:34252 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 14:24:59.011188 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:59.009919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.0.98.61:33659)
0620 14:24:59.010051 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:59.010056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:59.010159 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:24:59.010345 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:59.010349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:59.010406 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:59.010499 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:59.011066 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34266: BlockingRecv error: recv got EOF from 127.0.0.1:34266 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:24:59.310004 22890 catalog_manager.cc:1259] Loaded cluster ID: c8872f59647143c2a2b07647909d1e8d
I20260620 14:24:59.310086 22890 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0: loading cluster ID for follower catalog manager: success
I20260620 14:24:59.312021 22890 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0: acquiring CA information for follower catalog manager: success
I20260620 14:24:59.313256 22890 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a6a21e177b248548ac1cc71699979e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:24:59.326015 22889 catalog_manager.cc:1259] Loaded cluster ID: c8872f59647143c2a2b07647909d1e8d
I20260620 14:24:59.326125 22889 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd: loading cluster ID for follower catalog manager: success
I20260620 14:24:59.328075 22889 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd: acquiring CA information for follower catalog manager: success
I20260620 14:24:59.329388 22889 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12c35dd69f05418f98b5c74c3fd30abd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:24:59.529325 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:59.525310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.0.98.61:33659)
0620 14:24:59.525554 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:59.525560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:59.525622 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:24:59.525808 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:59.525812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:59.525948 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:59.526068 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:59.526337 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.526881 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.527791 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.528018 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.528127 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.528458 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:59.528895 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:59.529034 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:59.529078 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:24:59.529147 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260620 14:24:59.535701 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:59.532419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.0.98.60:34701)
0620 14:24:59.532613 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:59.532617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:59.532629 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:24:59.532705 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:59.532707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:59.532826 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:59.532924 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:59.533098 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.533604 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.534362 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.534587 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.534661 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.534901 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:59.535310 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:59.535397 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:59.535451 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:24:59.535495 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260620 14:24:59.542404 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:59.538580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.0.98.60:34701)
0620 14:24:59.538983 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:59.538987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:59.539042 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:24:59.539173 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:59.539176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:59.539233 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:59.539302 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:59.539558 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.540163 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.541236 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.541567 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.541664 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.541927 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:59.541992 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:59.542073 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:59.542183 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:24:59.542252 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260620 14:24:59.543139 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:59.538532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.0.98.61:33659)
0620 14:24:59.538648 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:59.538652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:59.539148 (+ 496us) server_negotiation.cc:407] Connection header received
0620 14:24:59.539312 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:59.539317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:59.539371 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:59.539444 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:59.539777 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.540581 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.541847 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.542170 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.542280 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.542647 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:59.542739 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:59.542784 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:59.542947 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:24:59.543009 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:24:59.543509 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:59.538519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.0.98.62:33473)
0620 14:24:59.539060 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:59.539065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:59.539077 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:24:59.539194 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:59.539196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:59.539355 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:59.539581 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:59.539794 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.540581 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.541811 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.542163 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.542514 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.542771 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:59.543123 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:59.543238 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:59.543306 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:24:59.543363 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":58,"threads_started":1}
I20260620 14:24:59.546384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 22638
I20260620 14:24:59.552349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 22713
I20260620 14:24:59.558092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 22788
2026-06-20T14:24:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3124 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:59 dist-test-slave-87nc krb5kdc[23005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:59 dist-test-slave-87nc krb5kdc[23005](info): set up 2 sockets
Jun 20 14:24:59 dist-test-slave-87nc krb5kdc[23005](info): commencing operation
krb5kdc: starting...
W20260620 14:25:00.612197 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/test-user.keytab.
Jun 20 14:25:00 dist-test-slave-87nc krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:25:00.736745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46119
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46119,127.0.98.61:40607,127.0.98.60:35609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:00.846294 23018 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:25:00.846591 23018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:00.846614 23018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:00.846716 23018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:00.850848 23018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:00.850924 23018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:00.850943 23018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:00.851011 23018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:00.851037 23018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:00.856806 23018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46119,127.0.98.61:40607,127.0.98.60:35609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:00.857954 23018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:00.859011 23018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:00.867875 23024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:00.868068 23018 server_base.cc:1034] running on GCE node
W20260620 14:25:00.867887 23023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:00.867887 23026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:00.868901 23018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:00.870357 23018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:00.871590 23018 hybrid_clock.cc:648] HybridClock initialized: now 1781965500871579 us; error 33 us; skew 500 ppm
Jun 20 14:25:00 dist-test-slave-87nc krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:00.877471 23018 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:25:00.878403 23018 webserver.cc:460] Webserver started at http://127.0.98.62:34905/ using document root <none> and password file <none>
I20260620 14:25:00.878942 23018 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:00.879040 23018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:00.879238 23018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:00.881340 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/data/instance:
uuid: "6176febdd2014e3cb5efaeadb251512b"
format_stamp: "Formatted at 2026-06-20 14:25:00 on dist-test-slave-87nc"
server_key: "16d850130b0f44179a4ba10883ab03c2"
server_key_iv: "c7ad1064cc6e019a086eea51c59c3cdc"
server_key_version: "encryptionkey@0"
I20260620 14:25:00.881842 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/wal/instance:
uuid: "6176febdd2014e3cb5efaeadb251512b"
format_stamp: "Formatted at 2026-06-20 14:25:00 on dist-test-slave-87nc"
server_key: "16d850130b0f44179a4ba10883ab03c2"
server_key_iv: "c7ad1064cc6e019a086eea51c59c3cdc"
server_key_version: "encryptionkey@0"
I20260620 14:25:00.885535 23018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:25:00.888015 23033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:00.889240 23018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:25:00.889355 23018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/wal
uuid: "6176febdd2014e3cb5efaeadb251512b"
format_stamp: "Formatted at 2026-06-20 14:25:00 on dist-test-slave-87nc"
server_key: "16d850130b0f44179a4ba10883ab03c2"
server_key_iv: "c7ad1064cc6e019a086eea51c59c3cdc"
server_key_version: "encryptionkey@0"
I20260620 14:25:00.889441 23018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:00.905015 23018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:00.907989 23018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:00.908154 23018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:00.915088 23018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46119
I20260620 14:25:00.915423 23085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46119 every 8 connection(s)
I20260620 14:25:00.916112 23018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:00.918850 23086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:00.922580 23086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:25:00.923228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 23018
I20260620 14:25:00.923377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:00.923627 392 external_mini_cluster.cc:1428] Setting key 3cf27a3921256e3db0618b22a98129e8
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 14:25:00 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:00.940948 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:00.925876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.0.98.62:46119)
0620 14:25:00.926189 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:00.926201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:00.926246 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:25:00.926790 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:00.926799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:00.927052 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:00.927273 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:00.928164 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.929698 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.930701 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.931405 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.931588 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:00.935895 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:00.935905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:00.935923 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:00.935958 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:00.937688 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:00.938173 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:00.938179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:00.938181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:00.938305 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:00.938622 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:00.938625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:00.938627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:00.939028 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:00.939174 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:00.939543 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:25:00.939847 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260620 14:25:00.940948 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:00.924983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46119 (local address 127.0.0.1:35222)
0620 14:25:00.925487 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:00.925541 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:00.926476 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:00.927267 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:00.927285 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:00.927686 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:00.928005 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:00.928020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.929829 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.929832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:00.930575 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:00.930584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.931571 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.931573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:00.931652 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:00.932363 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:00.932380 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:00.935649 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:00.937794 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:00.937802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:00.937810 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:00.938035 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:00.938400 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:00.938405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:00.938407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:00.938523 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:00.939143 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:00.939150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:00.939303 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 14:25:00.939471 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 14:25:00.939893 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":136,"spinlock_wait_cycles":4352,"thread_start_us":140,"threads_started":1}
I20260620 14:25:00.943334 23086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:25:00.944178 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:00.943905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40607 (local address 127.0.0.1:42520)
0620 14:25:00.944049 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:00.944116 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 14:25:00.944409 23036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
I20260620 14:25:00.945266 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:00.945060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40607 (local address 127.0.0.1:42526)
0620 14:25:00.945163 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:00.945213 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:25:00.945446 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40607: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:25:00.951133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40607
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46119,127.0.98.61:40607,127.0.98.60:35609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:25:00.994652 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } attempt: 1
I20260620 14:25:00.995563 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:00.995279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40607 (local address 127.0.0.1:42538)
0620 14:25:00.995426 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:00.995488 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:25:00.996304 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:00.996099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40607 (local address 127.0.0.1:42540)
0620 14:25:00.996192 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:00.996236 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:25:00.996457 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40607: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
W20260620 14:25:01.060859 23093 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:25:01.061162 23093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:01.061195 23093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:01.061234 23093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:25:01.064661 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } attempt: 2
I20260620 14:25:01.065500 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.065201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40607 (local address 127.0.0.1:42544)
0620 14:25:01.065366 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.065427 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 14:25:01.065863 23093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:01.065960 23093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:01.065999 23093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:01.066025 23093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:01.066057 23093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:01.066277 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.066064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40607 (local address 127.0.0.1:42558)
0620 14:25:01.066175 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.066220 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:25:01.066462 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40607: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40607: connect: Connection refused (error 111)
I20260620 14:25:01.071664 23093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46119,127.0.98.61:40607,127.0.98.60:35609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:01.072794 23093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:01.073843 23093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:01.082348 23101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:01.082336 23098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:01.082531 23093 server_base.cc:1034] running on GCE node
W20260620 14:25:01.082319 23099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:01.083184 23093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:01.084336 23093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:01.085526 23093 hybrid_clock.cc:648] HybridClock initialized: now 1781965501085511 us; error 28 us; skew 500 ppm
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:01.088300 23093 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:25:01.089113 23093 webserver.cc:460] Webserver started at http://127.0.98.61:44875/ using document root <none> and password file <none>
I20260620 14:25:01.089605 23093 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:01.089660 23093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:01.089829 23093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:01.091882 23093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/data/instance:
uuid: "65e148f2819e492098bb3aeb5d69b99d"
format_stamp: "Formatted at 2026-06-20 14:25:01 on dist-test-slave-87nc"
server_key: "6bd78401e19874561defd6b66515825e"
server_key_iv: "f65d0bbd8da0046d14ee937c8e0850bf"
server_key_version: "encryptionkey@0"
I20260620 14:25:01.092392 23093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/wal/instance:
uuid: "65e148f2819e492098bb3aeb5d69b99d"
format_stamp: "Formatted at 2026-06-20 14:25:01 on dist-test-slave-87nc"
server_key: "6bd78401e19874561defd6b66515825e"
server_key_iv: "f65d0bbd8da0046d14ee937c8e0850bf"
server_key_version: "encryptionkey@0"
I20260620 14:25:01.095948 23093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:25:01.098218 23108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:01.099386 23093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:01.099479 23093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/wal
uuid: "65e148f2819e492098bb3aeb5d69b99d"
format_stamp: "Formatted at 2026-06-20 14:25:01 on dist-test-slave-87nc"
server_key: "6bd78401e19874561defd6b66515825e"
server_key_iv: "f65d0bbd8da0046d14ee937c8e0850bf"
server_key_version: "encryptionkey@0"
I20260620 14:25:01.099561 23093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:01.123095 23093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:01.125902 23093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:01.126065 23093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:01.133219 23093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40607
I20260620 14:25:01.133509 23160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40607 every 8 connection(s)
I20260620 14:25:01.134207 23093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:25:01.136894 23161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:01.137555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 23093
I20260620 14:25:01.137647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-1/wal/instance
I20260620 14:25:01.137919 392 external_mini_cluster.cc:1428] Setting key 41fdae2bcbb25e7c37c5fc9c4f3fa874
I20260620 14:25:01.140623 23161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:01.155045 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.142503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.0.98.62:46119)
0620 14:25:01.142708 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.142715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.143482 (+ 767us) server_negotiation.cc:407] Connection header received
0620 14:25:01.144472 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.144478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.144538 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.144654 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.145773 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.146335 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.147571 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.147850 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.147971 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.150898 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.150905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.150911 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.150944 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.152721 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.153206 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.153213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.153217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.153318 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.153712 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.153717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.153721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.153994 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.154102 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.154684 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.154835 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:25:01.155885 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.142848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46119 (local address 127.0.0.1:35224)
0620 14:25:01.143311 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.143362 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.144253 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.144782 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.144788 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.145115 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.145633 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.145647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.146474 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.146477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.147470 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.147479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.148037 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.148043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.148213 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.148840 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.148866 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.150728 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:01.152836 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.152840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.152853 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.153100 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.153445 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.153448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.153449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.153607 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.154127 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:01.154133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:01.154393 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 14:25:01.154578 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:25:01.154863 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":134,"threads_started":1}
I20260620 14:25:01.157493 23161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:25:01.157636 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } attempt: 3
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:25:01.166332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUQknKY/build/debug/bin/kudu
/tmp/dist-test-taskUQknKY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35609
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46119,127.0.98.61:40607,127.0.98.60:35609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:25:01.168561 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.158027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40607 (local address 127.0.0.1:42564)
0620 14:25:01.158440 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.158452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.158515 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.158982 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.158985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.159147 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.159671 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.159679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.160665 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.160667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.161272 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.161294 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.161842 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.161847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.161962 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.162593 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.162604 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.163915 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:01.166787 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.166789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.166791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.166980 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.167290 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.167292 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.167293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.167352 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.167933 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:01.167939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:01.168055 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 14:25:01.168179 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 14:25:01.168339 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338}
I20260620 14:25:01.168690 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.158104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.0.98.61:40607)
0620 14:25:01.158456 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.158470 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.158495 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:25:01.158587 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.158597 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.158819 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.159817 (+ 998us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.159908 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.160555 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.161397 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.161712 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.161859 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.164041 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.164049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.164059 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.164079 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.166680 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.167087 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.167096 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.167101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.167200 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.167431 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.167437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.167442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.167817 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.167959 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.168291 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.168471 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":246,"spinlock_wait_cycles":4352,"thread_start_us":128,"threads_started":1}
I20260620 14:25:01.170161 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.158006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40607 (local address 127.0.0.1:42566)
0620 14:25:01.158176 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.158211 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.158329 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.158928 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.158932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.159117 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.159280 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.159290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.160359 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.160363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.161057 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.161065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.161810 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.161815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.162028 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.162665 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.162676 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.164477 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:01.168572 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.168575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.168577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.168956 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.169201 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.169203 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.169206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.169280 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.169626 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:01.169628 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:01.169714 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 14:25:01.169824 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:25:01.169956 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":45}
I20260620 14:25:01.170176 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.158187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.0.98.61:40607)
0620 14:25:01.158704 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.158710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.158720 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:25:01.158747 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.158749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.158807 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.158913 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.159402 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.160241 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.161155 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.161693 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.161872 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.164628 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.164634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.164638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.164662 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.168471 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.169048 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.169050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.169052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.169125 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.169353 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.169356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.169357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.169516 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.169627 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.169865 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.170012 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":465,"spinlock_wait_cycles":5504,"thread_start_us":87,"threads_started":1}
I20260620 14:25:01.170542 23086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:25:01.171209 23161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:25:01.171437 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.171045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51444)
0620 14:25:01.171283 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.171348 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260620 14:25:01.172006 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.171668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51456)
0620 14:25:01.171838 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.171924 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:25:01.172190 23109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
I20260620 14:25:01.172379 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.172137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51470)
0620 14:25:01.172253 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.172315 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:25:01.172556 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35609: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
I20260620 14:25:01.173097 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.172910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51480)
0620 14:25:01.173011 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.173051 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:25:01.173323 23161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35609: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
I20260620 14:25:01.203732 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } attempt: 1
I20260620 14:25:01.204594 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.204295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51488)
0620 14:25:01.204446 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.204518 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:25:01.205344 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.205128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51500)
0620 14:25:01.205230 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.205275 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:25:01.205495 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35609: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
I20260620 14:25:01.222569 23161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } attempt: 1
I20260620 14:25:01.223416 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.223116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51510)
0620 14:25:01.223260 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.223340 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:25:01.224218 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.224005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51518)
0620 14:25:01.224110 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.224155 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:25:01.224361 23161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35609: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
W20260620 14:25:01.279385 23168 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:25:01.279660 23168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:01.279722 23168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:01.279855 23168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:25:01.280642 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } attempt: 2
I20260620 14:25:01.281417 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.281127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51520)
0620 14:25:01.281266 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.281344 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 14:25:01.282179 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.281991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51532)
0620 14:25:01.282083 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.282128 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:25:01.282327 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35609: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
W20260620 14:25:01.284361 23168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:01.284411 23168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:01.284451 23168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:01.284485 23168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:01.284521 23168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:01.289883 23168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46119,127.0.98.61:40607,127.0.98.60:35609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:01.290999 23168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:01.292109 23168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:25:01.292517 23161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } attempt: 2
I20260620 14:25:01.293344 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.293030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51542)
0620 14:25:01.293198 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.293266 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 14:25:01.294050 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.293827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51546)
0620 14:25:01.293955 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.293999 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:25:01.294173 23161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35609: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35609: connect: Connection refused (error 111)
W20260620 14:25:01.300930 23174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:01.300951 23175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:01.300937 23177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:01.301694 23168 server_base.cc:1034] running on GCE node
I20260620 14:25:01.302232 23168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:01.303346 23168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:01.304522 23168 hybrid_clock.cc:648] HybridClock initialized: now 1781965501304489 us; error 48 us; skew 500 ppm
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:01.307236 23168 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:25:01.308158 23168 webserver.cc:460] Webserver started at http://127.0.98.60:40913/ using document root <none> and password file <none>
I20260620 14:25:01.308701 23168 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:01.308764 23168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:01.309007 23168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:01.311088 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/data/instance:
uuid: "4661a1e5a0a54017993cabdec54d7e74"
format_stamp: "Formatted at 2026-06-20 14:25:01 on dist-test-slave-87nc"
server_key: "23fcfe64960c502709051f0c6ea1d8f8"
server_key_iv: "957c6fa487ca9420b038f064dd3c5b6d"
server_key_version: "encryptionkey@0"
I20260620 14:25:01.311663 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/wal/instance:
uuid: "4661a1e5a0a54017993cabdec54d7e74"
format_stamp: "Formatted at 2026-06-20 14:25:01 on dist-test-slave-87nc"
server_key: "23fcfe64960c502709051f0c6ea1d8f8"
server_key_iv: "957c6fa487ca9420b038f064dd3c5b6d"
server_key_version: "encryptionkey@0"
I20260620 14:25:01.315228 23168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:25:01.317677 23184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:01.318943 23168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:01.319085 23168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/wal
uuid: "4661a1e5a0a54017993cabdec54d7e74"
format_stamp: "Formatted at 2026-06-20 14:25:01 on dist-test-slave-87nc"
server_key: "23fcfe64960c502709051f0c6ea1d8f8"
server_key_iv: "957c6fa487ca9420b038f064dd3c5b6d"
server_key_version: "encryptionkey@0"
I20260620 14:25:01.319190 23168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:01.341779 23168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:01.345048 23168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:01.345279 23168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:01.352919 23168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35609
I20260620 14:25:01.353200 23236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35609 every 8 connection(s)
I20260620 14:25:01.353981 23168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:25:01.354281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUQknKY/build/debug/bin/kudu as pid 23168
I20260620 14:25:01.354404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUQknKY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965214882667-392-0/minicluster-data/master-2/wal/instance
I20260620 14:25:01.354733 392 external_mini_cluster.cc:1428] Setting key 09d6d44ebc267a0d232f3526448bf2d2
I20260620 14:25:01.357329 23237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:01.362140 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:25:01.369884 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.356119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.0.98.60:35609)
0620 14:25:01.356515 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.356530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.356565 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:25:01.357207 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.357223 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.357602 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.357876 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.358827 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.359830 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.361091 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.361554 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.361660 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.364666 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.364674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.364708 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.364738 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.366627 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.367216 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.367222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.367227 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.367374 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.367630 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.367633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.367635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.368105 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.368293 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.368580 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.368875 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":139,"threads_started":1}
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:01.374825 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.364433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.0.98.62:46119)
0620 14:25:01.364585 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.364590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.365021 (+ 431us) server_negotiation.cc:407] Connection header received
0620 14:25:01.365144 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.365151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.365216 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.365306 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.366183 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.366871 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.367921 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.368198 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.368310 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.371085 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.371089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.371092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.371122 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.373118 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.373618 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.373622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.373624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.373709 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.373954 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.373957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.373959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.374144 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.374247 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.374614 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.374707 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:25:01.374898 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.364343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46119 (local address 127.0.0.1:35226)
0620 14:25:01.364862 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.364916 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.365045 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.365376 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.365384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.365859 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.366037 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.366046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.366985 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.366987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.367765 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.367774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.368318 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.368321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.368440 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.369106 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.369123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.370943 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:01.373219 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.373228 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.373233 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.373536 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.373800 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.373802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.373803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.373873 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.374264 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:01.374271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:01.374398 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 14:25:01.374545 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 14:25:01.374716 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":104,"threads_started":1}
I20260620 14:25:01.375923 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:25:01.381507 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } attempt: 3
I20260620 14:25:01.382110 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.372251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.0.98.62:46119)
0620 14:25:01.372615 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.372628 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.372642 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:25:01.372680 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.372684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.372736 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.372842 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.373446 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.374180 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.375309 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.375628 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.375705 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.378398 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.378402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.378406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.378435 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.380200 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.380671 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.380674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.380675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.380747 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.381056 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.381059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.381060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.381229 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.381480 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.381599 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.381740 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260620 14:25:01.384749 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.376383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40607 (local address 127.0.0.1:42580)
0620 14:25:01.376629 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.376641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.376697 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.376946 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.376948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.377071 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.377154 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.377159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.377950 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.377954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.378664 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.378673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.379182 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.379185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.379272 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.379683 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.379697 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.381186 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:01.383097 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.383099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.383101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.383348 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.383668 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.383671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.383673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.383755 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.384258 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:01.384261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:01.384351 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 14:25:01.384457 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:25:01.384586 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260620 14:25:01.384766 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.376515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.0.98.61:40607)
0620 14:25:01.376667 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.376672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.376688 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:25:01.376768 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.376773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.376831 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.376922 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.377265 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.377851 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.378752 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.379054 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.379170 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.381319 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.381326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.381331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.381357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.382973 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.383459 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.383463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.383464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.383555 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.383906 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.383909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.383911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.384112 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.384187 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.384509 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.384622 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:25:01.385334 23161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } attempt: 3
I20260620 14:25:01.386188 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:25:01.392102 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.381888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51550)
0620 14:25:01.382006 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.382022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.382098 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.382456 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.382458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.382631 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.382720 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.382726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.384245 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.384249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.385137 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.385147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.385994 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.385997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.386084 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.386806 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.386820 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.388351 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:01.390305 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.390310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.390313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.390605 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.390950 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.390954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.390956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.391040 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.391482 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:01.391485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:01.391593 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 14:25:01.391697 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:25:01.391902 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
Jun 20 14:25:01 dist-test-slave-87nc krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:25:01.393776 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.382160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.0.98.60:35609)
0620 14:25:01.382272 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.382276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.382298 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:25:01.382328 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.382331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.382368 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.382445 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.382873 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.384129 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.385264 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.385637 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.385723 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.388491 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.388495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.388498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.388519 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.390177 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.390723 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.390726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.390728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.390818 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.391141 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.391145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.391148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.391342 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.391436 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.393440 (+ 2004us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.393580 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:25:01.396039 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.383648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42582 (local address 127.0.98.61:40607)
0620 14:25:01.383758 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.383793 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.383807 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:25:01.383888 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.383891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.383936 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.384018 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.384747 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.385487 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.386777 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.387138 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.387224 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.392122 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.392127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.392129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.392156 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.393750 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.394361 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.394364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.394366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.394453 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.394809 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.394815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.394818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.395055 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.395266 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.395663 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.395833 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:25:01.396289 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.385683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51566)
0620 14:25:01.385770 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.385786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.385855 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.386547 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.386550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.386727 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.386841 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.386851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.387871 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.387875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.388662 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.388675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.389238 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.389241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.389561 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.389990 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.390005 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.391467 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:01.394705 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.394708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.394709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.394957 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.395275 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.395281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.395285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.395372 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.395832 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:01.395835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:01.395923 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 14:25:01.396020 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:25:01.396159 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:25:01.397053 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.386626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35609 (local address 127.0.0.1:51568)
0620 14:25:01.386729 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.386747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.386808 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.388099 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.388102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.388274 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.388331 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.388335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.389812 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.389818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.390635 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.390644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.391056 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.391059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.391133 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.391969 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.391985 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.393434 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:01.395569 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.395575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.395577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.395837 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.396116 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:01.396118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:01.396120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:01.396198 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:01.396592 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:01.396596 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:01.396684 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 14:25:01.396761 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:25:01.396910 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"spinlock_wait_cycles":3456}
I20260620 14:25:01.397199 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.385907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.0.98.60:35609)
0620 14:25:01.386316 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.386325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.386339 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:25:01.386370 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.386373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.386411 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.386530 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.386927 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.387653 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.388762 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.389105 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.389234 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.391591 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.391594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.391597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.391626 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.394577 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.395069 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.395073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.395074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.395168 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.395473 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.395476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.395478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.395664 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.395745 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.396824 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.397012 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":336,"thread_start_us":119,"threads_started":1}
I20260620 14:25:01.397620 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.387046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.0.98.60:35609)
0620 14:25:01.387896 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.387906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.387919 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:25:01.387948 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.387951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.387993 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.388092 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.388435 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.389701 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.390733 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.390956 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.391057 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.393577 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.393580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.393584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.393612 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.395463 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.395937 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.395940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.395942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.396022 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.396297 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.396299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.396301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.396499 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.397298 (+ 799us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.397365 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.397465 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":99,"threads_started":1}
I20260620 14:25:01.397598 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b: Bootstrap starting.
I20260620 14:25:01.400127 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:25:01.400519 23086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:01.400758 23161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d: Bootstrap starting.
I20260620 14:25:01.400925 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74: Bootstrap starting.
I20260620 14:25:01.401401 23086 log.cc:824] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:01.401242 23060 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6176febdd2014e3cb5efaeadb251512b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230
I20260620 14:25:01.403148 23237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:01.403219 23161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:01.403920 23237 log.cc:824] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:01.404006 23161 log.cc:824] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:01.405987 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74: No bootstrap required, opened a new log
I20260620 14:25:01.405988 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b: No bootstrap required, opened a new log
I20260620 14:25:01.405988 23161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d: No bootstrap required, opened a new log
I20260620 14:25:01.409024 23086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } }
I20260620 14:25:01.409263 23086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6176febdd2014e3cb5efaeadb251512b, State: Initialized, Role: FOLLOWER
I20260620 14:25:01.409698 23086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } }
I20260620 14:25:01.409854 23161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } }
I20260620 14:25:01.410070 23161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e148f2819e492098bb3aeb5d69b99d, State: Initialized, Role: FOLLOWER
I20260620 14:25:01.410368 23161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } }
I20260620 14:25:01.410415 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.410578 23237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } }
I20260620 14:25:01.410807 23237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4661a1e5a0a54017993cabdec54d7e74, State: Initialized, Role: FOLLOWER
I20260620 14:25:01.410862 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:25:01.411124 23237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } }
I20260620 14:25:01.411196 23086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:01.411262 23060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:25:01.411381 23060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:01.412109 23161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:01.412281 23237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:01.412706 23060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } }
I20260620 14:25:01.414790 23135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6176febdd2014e3cb5efaeadb251512b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "65e148f2819e492098bb3aeb5d69b99d"
I20260620 14:25:01.415000 23135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:01.413712 23060 leader_election.cc:288] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [CANDIDATE]: Term 1 election: Requested vote from peers 65e148f2819e492098bb3aeb5d69b99d (127.0.98.61:40607), 4661a1e5a0a54017993cabdec54d7e74 (127.0.98.60:35609)
I20260620 14:25:01.416337 23135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6176febdd2014e3cb5efaeadb251512b in term 1.
W20260620 14:25:01.417143 23271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:25:01.417208 23271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:25:01.414855 23211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6176febdd2014e3cb5efaeadb251512b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4661a1e5a0a54017993cabdec54d7e74"
I20260620 14:25:01.418061 23211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:01.419366 23211 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6176febdd2014e3cb5efaeadb251512b in term 1.
I20260620 14:25:01.419806 23034 leader_election.cc:302] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4661a1e5a0a54017993cabdec54d7e74, 6176febdd2014e3cb5efaeadb251512b; no voters:
W20260620 14:25:01.420471 23282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:25:01.420543 23282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:25:01.422555 23168 master_runner.cc:263] No leader master found from master 4661a1e5a0a54017993cabdec54d7e74
I20260620 14:25:01.422618 23168 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:25:01.423846 23248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:01.423944 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.424074 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:25:01.424222 23248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [term 1 LEADER]: Becoming Leader. State: Replica: 6176febdd2014e3cb5efaeadb251512b, State: Running, Role: LEADER
I20260620 14:25:01.424280 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.424400 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:25:01.424422 23018 master_runner.cc:263] No leader master found from master 6176febdd2014e3cb5efaeadb251512b
I20260620 14:25:01.424474 23018 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:25:01.424747 23248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } }
I20260620 14:25:01.424942 23093 master_runner.cc:263] No leader master found from master 65e148f2819e492098bb3aeb5d69b99d
I20260620 14:25:01.425002 23093 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:25:01.426556 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6176febdd2014e3cb5efaeadb251512b. Latest consensus state: current_term: 1 leader_uuid: "6176febdd2014e3cb5efaeadb251512b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.426717 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:01.428416 23294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:01.429463 23294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 14:25:01.431113 23283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 14:25:01.432618 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.422551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.0.98.60:35609)
0620 14:25:01.422688 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.422692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.423227 (+ 535us) server_negotiation.cc:407] Connection header received
0620 14:25:01.423412 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.423416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.423459 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.423510 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.424601 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.425397 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.426596 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.426950 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.427024 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.428934 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.428937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.428940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.428965 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.430643 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.431175 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.431182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.431184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.431295 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.431614 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.431620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.431623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.431880 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.431986 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.432338 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.432460 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:25:01.436667 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.423293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.0.98.61:40607)
0620 14:25:01.423396 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.423400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.428085 (+ 4685us) server_negotiation.cc:407] Connection header received
0620 14:25:01.428197 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.428201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.428250 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.428325 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.429132 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.429841 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.430902 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.431213 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.431305 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.433114 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.433118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.433120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.433149 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.434833 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.435325 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.435329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.435330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.435408 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.435700 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.435703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.435704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.435946 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.436045 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.436403 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.436517 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:25:01.441730 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.427145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.0.98.62:46119)
0620 14:25:01.427246 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.427252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.433102 (+ 5850us) server_negotiation.cc:407] Connection header received
0620 14:25:01.433273 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.433276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.433327 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.433379 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.434135 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.434929 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.436037 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.436399 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.436474 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.438164 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:01.438169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:01.438171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:01.438201 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:01.439884 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.440359 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.440363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.440365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.440450 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:01.440748 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:01.440752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:01.440753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:01.440985 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:01.441084 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:01.441443 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 14:25:01.441571 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:25:01.431196 23283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:25:01.443976 23211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [term 1 FOLLOWER]: Refusing update from remote peer 6176febdd2014e3cb5efaeadb251512b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:01.444346 23135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [term 1 FOLLOWER]: Refusing update from remote peer 6176febdd2014e3cb5efaeadb251512b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:01.445043 23292 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:01.445218 23248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:01.449139 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6176febdd2014e3cb5efaeadb251512b. Latest consensus state: current_term: 1 leader_uuid: "6176febdd2014e3cb5efaeadb251512b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.449268 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:25:01.449537 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6176febdd2014e3cb5efaeadb251512b. Latest consensus state: current_term: 1 leader_uuid: "6176febdd2014e3cb5efaeadb251512b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.449666 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:25:01.454561 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6176febdd2014e3cb5efaeadb251512b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.454712 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:01.455909 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6176febdd2014e3cb5efaeadb251512b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.456027 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:25:01.456420 23297 mvcc.cc:205] Tried to move back new op lower bound from 7298930693904896000 to 7298930693838905344. Current Snapshot: MvccSnapshot[applied={T|T < 7298930693904896000}]
I20260620 14:25:01.460196 23298 mvcc.cc:205] Tried to move back new op lower bound from 7298930693904896000 to 7298930693838905344. Current Snapshot: MvccSnapshot[applied={T|T < 7298930693904896000}]
I20260620 14:25:01.460434 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6176febdd2014e3cb5efaeadb251512b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.460552 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:01.461519 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6176febdd2014e3cb5efaeadb251512b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6176febdd2014e3cb5efaeadb251512b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46119 } } peers { permanent_uuid: "65e148f2819e492098bb3aeb5d69b99d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40607 } } peers { permanent_uuid: "4661a1e5a0a54017993cabdec54d7e74" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35609 } } }
I20260620 14:25:01.461647 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:25:01.468242 23294 catalog_manager.cc:1347] Generated new cluster ID: 30b69a2b8a8a4b5894bf5965162134ef
I20260620 14:25:01.468324 23294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:01.503852 23294 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:01.505378 23294 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:01.515817 23294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6176febdd2014e3cb5efaeadb251512b: Generated new TSK 0
I20260620 14:25:01.516855 23294 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:25:01.553434 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.550808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.0.98.62:46119)
0620 14:25:01.550961 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.550966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.552350 (+ 1384us) server_negotiation.cc:407] Connection header received
0620 14:25:01.552555 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.552559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.552608 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.552660 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.553305 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35246: BlockingRecv error: recv got EOF from 127.0.0.1:35246 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 14:25:01.555094 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.551682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.0.98.61:40607)
0620 14:25:01.551823 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.551827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.554132 (+ 2305us) server_negotiation.cc:407] Connection header received
0620 14:25:01.554350 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.554353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.554401 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.554451 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.554976 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42600: BlockingRecv error: recv got EOF from 127.0.0.1:42600 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 14:25:01.557077 23240 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.555829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.0.98.60:35609)
0620 14:25:01.555964 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.555969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.556197 (+ 228us) server_negotiation.cc:407] Connection header received
0620 14:25:01.556398 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.556401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.556448 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.556498 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.556964 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51592: BlockingRecv error: recv got EOF from 127.0.0.1:51592 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 14:25:01.561533 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.560057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.0.98.61:40607)
0620 14:25:01.560157 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.560161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.560607 (+ 446us) server_negotiation.cc:407] Connection header received
0620 14:25:01.560830 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.560833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.560906 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.560957 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.561432 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42610: BlockingRecv error: recv got EOF from 127.0.0.1:42610 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 14:25:01.566351 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.564936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.0.98.61:40607)
0620 14:25:01.565030 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.565034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.565538 (+ 504us) server_negotiation.cc:407] Connection header received
0620 14:25:01.565705 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.565708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.565754 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.565813 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.566262 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42622: BlockingRecv error: recv got EOF from 127.0.0.1:42622 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 14:25:01.571624 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.570203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.0.98.61:40607)
0620 14:25:01.570304 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.570308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.570816 (+ 508us) server_negotiation.cc:407] Connection header received
0620 14:25:01.571007 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.571010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.571054 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.571101 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.571536 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42632: BlockingRecv error: recv got EOF from 127.0.0.1:42632 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 14:25:01.579026 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.577556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.0.98.61:40607)
0620 14:25:01.577674 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.577679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.578165 (+ 486us) server_negotiation.cc:407] Connection header received
0620 14:25:01.578347 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.578351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.578398 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.578445 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.578925 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42638: BlockingRecv error: recv got EOF from 127.0.0.1:42638 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 14:25:01.590991 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.589316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.0.98.61:40607)
0620 14:25:01.589429 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.589433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.590039 (+ 606us) server_negotiation.cc:407] Connection header received
0620 14:25:01.590245 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.590249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.590301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.590354 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.590885 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42652: BlockingRecv error: recv got EOF from 127.0.0.1:42652 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 14:25:01.611651 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.609755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.0.98.61:40607)
0620 14:25:01.609900 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.609904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.610547 (+ 643us) server_negotiation.cc:407] Connection header received
0620 14:25:01.610770 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.610774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.610860 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.610923 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.611525 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42664: BlockingRecv error: recv got EOF from 127.0.0.1:42664 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 14:25:01.647256 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.646161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.0.98.61:40607)
0620 14:25:01.646299 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.646303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.646446 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:25:01.646539 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.646542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.646600 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.646705 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.647143 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42680: BlockingRecv error: recv got EOF from 127.0.0.1:42680 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 14:25:01.714883 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.713579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.98.61:40607)
0620 14:25:01.713743 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.713750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.713961 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:25:01.714149 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.714153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.714213 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.714313 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.714767 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42688: BlockingRecv error: recv got EOF from 127.0.0.1:42688 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 14:25:01.846571 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.845341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.98.61:40607)
0620 14:25:01.845510 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.845515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.845642 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:25:01.845821 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.845825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.845883 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.845979 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.846449 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42698: BlockingRecv error: recv got EOF from 127.0.0.1:42698 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 14:25:02.105854 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:02.104700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.98.61:40607)
0620 14:25:02.104860 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:02.104864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:02.104969 (+ 105us) server_negotiation.cc:407] Connection header received
0620 14:25:02.105135 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:02.105138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:02.105193 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:02.105281 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:02.105728 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42704: BlockingRecv error: recv got EOF from 127.0.0.1:42704 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:25:02.421737 23282 catalog_manager.cc:1259] Loaded cluster ID: 30b69a2b8a8a4b5894bf5965162134ef
I20260620 14:25:02.421828 23282 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d: loading cluster ID for follower catalog manager: success
I20260620 14:25:02.423643 23282 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d: acquiring CA information for follower catalog manager: success
I20260620 14:25:02.424810 23282 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65e148f2819e492098bb3aeb5d69b99d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:25:02.444139 23283 catalog_manager.cc:1259] Loaded cluster ID: 30b69a2b8a8a4b5894bf5965162134ef
I20260620 14:25:02.444222 23283 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74: loading cluster ID for follower catalog manager: success
I20260620 14:25:02.446276 23283 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74: acquiring CA information for follower catalog manager: success
I20260620 14:25:02.447388 23283 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4661a1e5a0a54017993cabdec54d7e74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:25:02.623942 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:02.619913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.0.98.61:40607)
0620 14:25:02.620164 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:02.620172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:02.620220 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:25:02.620414 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:02.620422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:02.620554 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:02.620677 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:02.620963 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.621497 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.622387 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.622621 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.622738 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.623076 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:02.623507 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:02.623652 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:02.623688 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 14:25:02.623748 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":64,"threads_started":1}
I20260620 14:25:02.633275 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:02.628514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.0.98.60:35609)
0620 14:25:02.628843 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:02.628848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:02.628864 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:25:02.629088 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:02.629092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:02.629268 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:02.629396 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:02.629737 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.630592 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.631792 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.632045 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.632142 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.632446 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:02.632886 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:02.632977 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:02.633029 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:25:02.633078 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
I20260620 14:25:02.640837 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:02.636406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.0.98.60:35609)
0620 14:25:02.636559 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:02.636564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:02.636963 (+ 399us) server_negotiation.cc:407] Connection header received
0620 14:25:02.637060 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:02.637062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:02.637097 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:02.637155 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:02.637399 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.638105 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.639069 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.639413 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.639527 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.639891 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:02.640049 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:02.640148 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:02.640439 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 14:25:02.640570 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:25:02.641172 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:02.636653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.0.98.61:40607)
0620 14:25:02.636789 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:02.636793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:02.637560 (+ 767us) server_negotiation.cc:407] Connection header received
0620 14:25:02.637651 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:02.637654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:02.637702 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:02.637772 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:02.638003 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.638725 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.639882 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.640214 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.640298 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.640662 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:02.640755 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:02.640801 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:02.640990 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 14:25:02.641045 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:25:02.641660 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:02.636044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.0.98.62:46119)
0620 14:25:02.636824 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:02.636829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:02.636842 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:25:02.636928 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:02.636931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:02.637101 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:02.637244 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:02.637530 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.638297 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.639373 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.639736 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.639892 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.640288 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:02.640690 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:02.640786 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:02.641477 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 14:25:02.641539 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":66,"threads_started":1}
I20260620 14:25:02.644745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 23018
I20260620 14:25:02.651055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 23093
I20260620 14:25:02.656880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUQknKY/build/debug/bin/kudu with pid 23168
2026-06-20T14:25:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3098 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6222 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287767 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST